Henry and I are joined by the great Robert Wright: journalist, host of the Nonzero podcast, and author of many modern classics, including some of my favourites, such as The Moral Animal (still one of the best introductions to evolutionary psychology going) and Nonzero, a very interesting account of the biological and cultural evolution of cooperation and complexity.
His new book, The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning, combines an excellent, highly accessible, up-to-date introduction to modern AI with original, provocative arguments about what the technology means, both politically and metaphysically. Most commentators sit somewhere on a line running from “AI is overhyped nonsense” to “AI is radically transformative.” Bob goes much further: he thinks rapid advances in AI confront our species with something like a moral test of the kind a God might set, which we can only pass by overcoming our evolved tribal biases and becoming a cohesive global community.
We had some questions! Among many other topics, we discuss:
What Wright means by a “God test”
Whether AI is a god test on the scale of nuclear weapons, and why the coordination and verification problem is harder this time
How large language models (LLMs) come to represent meaning, and the case for modern AI as “reverse engineering” the mind
The sceptic’s case against imminent transformative AI: jaggedness, sample inefficiency, disembodiment and continual learning, and why I (Dan) hold longer timelines than Wright
Evolutionary psychology versus the blank slate: why Wright expects jagged rather than general intelligence, with a detour through Richard Sutton, Skinner and the “bitter lesson”
Technological determinism and agency: why true agency may be impossible inside competitive environments
The default trajectory: destabilisation, authoritarianism, and why a breakneck race to superintelligence could bring authoritarianism to America “through the back door”
The US and China race: steelmanning Dario Amodei’s arguments, the Superintelligence Strategy paper, and Wright’s contrarian (in America, at least) reading of Chinese intentions
The security dilemma and threat inflation: why defensively motivated moves get read as aggression, from the First World War to the South China Sea
Negativity bias and the discourse: my challenge that commentary on AI, and the book itself, tilts too far towards negativity, alarmism, and catastrophism
Does evolution have a purpose? Smolin’s cosmological natural selection and William Hamilton on directionality
Consciousness and epiphenomenalism
Links
Robert Wright’s Nonzero (newsletter and podcast)
The God Test (Simon & Schuster)
Mentioned in the episode:
“AI as Normal Technology”, Arvind Narayanan and Sayash Kapoor
Allan Dafoe on the 80,000 Hours podcast, on why technology is unstoppable and how to shape AI development anyway
“Machines of Loving Grace”, Dario Amodei
“Superintelligence Strategy”, Dan Hendrycks, Eric Schmidt and Alexandr Wang
Transcript
Please note that this transcript is lightly AI-generated and may contain minor mistakes.
Dan: Okay, welcome back. I’m Dan Williams, back with my co-host, Henry Shevlin, and today we’re joined by a true legend, the great Robert Wright. Bob is the author of the Nonzero newsletter, host of the Nonzero podcast, and author of some genuine modern classics, including some of my personal favourites, like The Moral Animal, which is one of the first and, I still think, one of the best introductions to evolutionary psychology, as well as Nonzero, which is a kind of sweeping story about the biological and cultural evolution of cooperation and complexity. More recently, he’s the author of The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning, which is going to be the topic of our conversation today. So Bob, welcome to the podcast.
Bob: Thanks for having me.
The God test and the cosmic frame
Dan: In terms of a continuum of views about the significance of AI, at one end you’ve got the people who say it’s all just a con, it’s hype, it doesn’t really work. Then you’ve got the people who say it’s an important technology, but fundamentally just another technology. Then you’ve got the people who say, no, this is radically transformative, we’re building potentially superintelligent agents that are going to completely upend the social order. And then we’ve got you, saying even more significantly than that, this is a God test and the grounds for a cosmic reckoning. So firstly, what do you mean by that? And secondly, why should we think it’s such a big deal?
Bob: I think it’s going to be, first of all, a total earthquake. A force of great magnitude is being unleashed. In that sense I’m on the same page with some other people, including, I would say, both doomers and some accelerationists who, for reasons I don’t understand, think that although we’re going to enter a singularity beyond which nothing is clear, everything’s going to work out fine. That would make more sense to me if they said they were religious and had religious faith in it. Otherwise I do not understand the basis for the optimism. It’s true that in a sense I step back further than most people in both camps in terms of how cosmically I locate our situation.
On the one hand, in the book, the thing I’m probably proudest of is trying to make clear, especially to a non-technical audience, what the secret sauce of the deep learning revolution is: why the capabilities of AI have grown so fast and will keep growing, and what is going on inside a large language model, to the extent that we can tell. So there are some very mundane aspirations. But it’s true that I make a show early on of saying I’m going to step back and view this in very cosmic terms, in the context of the whole three-and-a-half-billion-year history of life. And I do think, and some others would agree with this part, that if you ask Eliezer Yudkowsky whether the birth of silicon intelligence on this planet deserves an asterisk, even in the context of all of life, an asterisk that very few others rank with, he’d say yes. So I’m not uniquely cosmic in the way I look at this.
Now, when you start talking about what I mean by the God test, I should say the most important meaning of that is metaphorical. I’m always happy to talk about the possibility that some larger purpose is unfolding on the planet, and I do to some extent, largely in the appendix. But the main meaning of the term is that I think artificial intelligence poses us as a species with the kind of test that God would give, the kind God gives in the Bible: salvation is possible, but you’ve got to get your act together. You see things like that in the Old Testament and the New Testament, and in other religions. It’s a moral test. I think our species needs what you could call a moral upgrade, because I don’t think we can navigate the AI revolution successfully except as a cohesive global community, and we are a long way from being that. In order to become that, we’re going to have to become in some sense a better species morally, and I mean something a little less ambitious by that than it sounds. It’s not entirely impossible that we could succeed. But that gives you some sense of why the word cosmic is in the subtitle.
Is this like nuclear weapons?
Henry: Just a very quick clarification. I’m reminded of nuclear weapons here as another possible God test. We had this period where suddenly we had the capability to at least destroy civilisation, if not our species. Do you think it’s fair to call nuclear weapons a God test we’ve already, to some extent, passed? Or is this a different order of magnitude of test?
Bob: I think it’s quite different. There are parallels. One thing I’ve heard people say about AI is, hey, we can do this thing with China, because with nuclear weapons we were adversaries, if not enemies, with the Soviet Union, but we worked out some arms control deals. That’s true. But I think the degree of international coordination, even international governance, that’s going to be required this time, and the nature of it, mean you’re just going to have to be on better terms with the other countries than we were with the Soviet Union. This is not a few isolated arms control agreements. First of all, as people have noted, with AI you’re generally talking about a bigger verification challenge. The degree of monitoring, if you want to monitor big training runs, they’re pretty conspicuous, so it’s kind of doable. But even now, some people are starting to say, and Anthropic hinted at it in a paper they wrote about recursive self-improvement, that maybe at some point we should talk about a pause, or a slowdown, and if so, that will need to be globally coordinated. That’s fine, but with the current politics of the United States it’s hard for me to imagine that happening unless our relationship with China changes, because there will just be too many people saying, they say they’re pausing. We’ve already seen how, even though we shut off their supply of great microchips, they’re doing pretty incredible stuff. I guess they’re doing it with something a little less conspicuous than Elon’s Colossus Two in Tennessee. So I could go on, but ultimately it goes beyond a couple of specific quasi arms control agreements, and even those are in some cases more challenging to enforce than a nuclear treaty.
Why so bullish on the current paradigm?
Dan: I really want to get into the geopolitics and the arguments about international coordination, and also the cosmic framing. But we’re going to have some people listening or watching who are more sceptical of the technology itself. It would be helpful, Bob, if you could say a little about why you think, not just in principle we could build AI systems that substitute for human labour across the board and reach something like AGI and maybe artificial superintelligence, but why you’re so bullish on the current paradigm.
Bob: Okay. I gather you have two kinds of audience, audio and video. This won’t surprise the video audience, but maybe it’ll surprise the audio audience: I was a journalist in 1983. I’m pretty old. I interviewed Geoffrey Hinton for a piece I was writing on AI. Since then I’ve intermittently written about technology, not exclusively, but I’ve kept up with it. After the large language model got my attention, and it was the first time AI really got my attention in a big way, I went back and read the piece I’d written. I’d written the Time magazine cover story when the IBM computer beat Garry Kasparov for the world chess championship, and honestly I thought that was such a boring story that I steered it into a story about consciousness. I just didn’t think it was that interesting what the machine was doing, because it wasn’t very much like what a human mind does.
But back to Hinton. When GPT-3.5 and then 4 got my attention, I went back and read that piece, which did talk about his maverick approach. The term neural networks is in the piece, a favourite phrase of his at the time, along with massive parallelism. But I realised I really did not understand the potential of the paradigm. I didn’t understand the paradigm. My example of a neural network was kind of legit, because it came from a guy who had co-authored a paper with Hinton, but he was a psychologist who meant something different by neural network than what was most important to Hinton. The neural network I described was in theory going to be equipped with the ability to process language. But with that kind of network, where each node was going to represent a certain sense of a word, humans would have to implant in the machine the human understanding of the connection between the meaning and the words. You’d have to somehow, maybe you could automate it by feeding a dictionary into it, but somehow the explicit connection between individual words, or their specific senses, and meaning would have to be engineered. Humans would have to come up with a way of representing meaning and put it in the machine.
When I went back in 2023 and looked at a lecture Hinton had given in 2018, and started looking at how modern neural networks actually work, I realised we don’t have to put the meaning in. You just make them better at a task. It could be next word prediction, fill in the blank, translation into a foreign language. But all of these tasks, from the machine’s point of view, are just: here’s a bunch of gibberish, figure out the next gibberish. We’re not telling it that the symbols mean this or that, and yet, because we give it a broad framework for encoding each word at the beginning, we say, look, each word, and this is a slight oversimplification, should be a bunch of numbers separated by commas, whatever numbers you want, just keep changing the numbers until you finally get good at the task. So there’s no point where we’re telling the machine what the meaning of the words is, and yet we now understand that the machines wind up using the numbers as a means of representing the meaning of words.
Another slight oversimplification: imagine a graph with two axes. You’re plotting, say, animals, with velocity on one axis and lethality on the other. Tigers would be high on both, rattlesnakes high on one and not so high on the other. That’s two dimensions. In effect these machines, in an implicit way I won’t get into, are using a bunch of numbers to represent the meaning of words in a high-dimensional space. And it turns out, as you might expect, that words close in meaning are close to one another in that space. So there is this mapping onto meaning, and there’s actually a second kind that gets done with multimodal.
That was a mind-blower for me. This would save you some time if you didn’t really have to say much about the meaning of the words to the computer, or how it should represent that meaning, aside from saying it has to have some connection to these numbers. And then I realised you can apply this everywhere. It’s the same way Elon’s cars learn to drive: you give it human-like input data and then the output data that humans would provide. With Elon, the input data is visual and the output data is turn the steering wheel slightly to the right. And you can get the machine to, I think a fair way to put it, though not everyone in the field would agree, reverse engineer the functional components of the human mind. They don’t work exactly the way those components work, but the functionality, such as the representation of meaning, or in the case of image recognition something as fine-grained as edge detection neurons, gets reverse engineered. These machines, I’d say, reverse engineer an edge detection mechanism comparable to edge detection neurons.
I honestly think, even in the AI community, one reason I’m proud of some of the exposition in the book is that I’ve spent my whole life as a journalist trying to make things accessible to a lay audience, and experts are just inherently not good at this. It goes with the territory: when you know a lot about a field, you lose track of what the average person doesn’t know. If I ask why more people in the field don’t say, look, these things can reverse engineer any part of the human cognitive or perceptual apparatus, and all we have to do is feed in the data, why don’t they put it that simply, I think in part there are some disagreements, some people would quibble with my saying that, but in part it’s just that they’re experts, and there’s a lot they take for granted that would help people understand the magnitude of what’s unfolding, and they just don’t put it that way.
Semantics, syntax, and “normal technology”
Henry: The description of semantics you’re giving reminds me of an expression by the philosopher John Haugeland, who said, contrary to the view that you need to program in the meaning of every symbol, the symbol grounding view, if you take care of the syntax, the semantics takes care of itself. He was saying that back in the 1980s, and I think that captures what we’ve seen: that models, given enough data...
Bob: Well, that was early. Was he talking about neural networks?
Henry: He was expressing the hope that we wouldn’t need to do this deep, elaborate symbol grounding for every symbol in a system. So he was offering an optimistic picture of how things would work out. But just to press you a little further. Dan and I are completely aligned on the incredible linguistic capabilities of LLMs, and to be honest we’re probably aligned about the broadly transformative potential of AI too. But we’re also both fans of this “AI as normal technology” view that we’ve discussed on the show a few times, which says: yes, AI is a huge deal, it’s going to be one of the biggest technologies in human history, but it’s not unprecedented. It’s up there with the internet, or electrification, or maybe the industrial revolution. Your view is that it’s more than that. So what makes it different from these other big technological paradigm shifts?
Bob: First I’d say it’s hard to separate some of these things. You can’t imagine modern AI without the internet, because it trains on the internet. But that said, the advent of a new kind of intelligence that is not organically based and is as smart as the smartest organically based intelligence, that’s a first. Life has been here three and a half billion years. That’s a first. Secondly, to look at jobs, for example, there’s never been a technology where it was so at least plausible, we don’t know yet, but plausible, that this time, when it takes the jobs, there won’t be other jobs. It’s an open question. I can imagine scenarios, but it’s never been such a good question whether there will be anything for us to do.
And if you go down the line of other areas where it will have a huge impact: jobs is one, interpersonal relations is another, friendships, romance. Parents are going to be freaking out more and more about why their kid is spending so much time with the machine and less time with human friends. I’m not saying we won’t get through any of this, or all of it, eventually. I’m just saying the combined impact will be unprecedented, especially on such a short time scale. To some extent that’s just a function of how things happen these days. In my book Nonzero I spent a lot of time on the printing press, which was in many ways analogous to, well, narrowcasting, the internet, social media. But that took a long time. It took decades before you had actual wars plausibly attributable to the printing press. This is going to happen very fast, and along many dimensions. I think it’s going to be geopolitically destabilising, politically destabilising. We could talk about the persuasive capabilities of these things. When you add it all up, the net impact is going to exceed, certainly on this time scale, anything in history. That’s why I’d like to see us slow down and proceed more deliberately, and as soon as possible proceed as a global community, and recognise that this is a challenge we can only respond to adroitly as a global community. I’d like to hear more about your side of the argument. You can define normal in a way that would make this work; if you define normal as abnormal, I’m on board. But what’s the definition you’re using?
Henry: I’m going to pass to Dan, because in our debates on this I’ve been more on the transformative side and Dan’s been more on the normal technology side. Right, Dan?
Dan: To an extent. The way I’d put it is, once we’ve got AIs that are smarter than human beings across the board, then yes, I agree with you, Bob, that’s not a normal technology. That’s, to put it mildly, a unique, sui generis technology. But I still think there’s a sceptical position here that says, why should we think we’re on track over the next several years to get to that point with the current AI paradigm? You do a really great job in the book showing that there’s a really important sense in which LLMs genuinely understand meaning and are really intelligent in lots of ways. But a sceptic is going to say, okay, but there are still so many things they can’t do. They can’t do continual learning the way non-human animals and human beings can. They’re extremely sample inefficient, they need an enormous amount of data to learn various capabilities. They’re disembodied. They’ve got this jagged competence profile where they’re superhuman at some things and really kind of stupid at others. So a lot of what you’re arguing in the book is not just that the current paradigm is really impressive and incredibly intelligent along specific dimensions, it’s that we are on the cusp, over the next several years, of reaching truly transformative AI. I take that seriously as a possibility, but I don’t think it’s the most likely scenario. I think I’ve got longer timelines than you do, Bob. So what’s the case for thinking this is likely to happen over the next five or ten years, rather than more gradually over the next several decades?
Bob: Well, first, you’ve pointed to something they’re not good at, which is real-time, continual learning, the way a worker learns on the job, the way we all learn things through our lives. I’d also say that a few years ago you could have pointed to some things people were laughing at AIs for, and they’re laughing less now. Reasoning is one, and that’s pretty big. You can still find it making some stupid reasoning errors, but the whole chain-of-thought reasoning thing was a true and very important innovation that just wasn’t there three years ago. That’s super powerful. And relatedly, training techniques have got refined in what’s called post-training. By the way, the remarkable thing about this vocabulary, and maybe they’ve changed it, but as of a year or two ago, there was such a thing as pre-training and such a thing as post-training, but no training. And I’m thinking, wait, the people who came up with this are our leading lights? We’re trusting them to guide us through this revolution? I just want to put a wall around Silicon Valley. You want to build machines, fine, but somebody else should be in charge of making sense of this, if that’s your approach to coherent terminology. I’m joking. But there have been real refinements in training that have added real dimensions of capability.
I want to get back to the jaggedness. You said it’s disembodied. Well, yes, but again, the basic approach to reverse engineering parts of the mind, and I will defend that metaphor, can be applied to any sensory channel: tactile, olfactory, visual. And there is real progress in robotics. Some things will take longer than others, but I don’t see a particular place where there’s a super fundamental obstacle ahead. Now, as for continual learning and, relatedly, the jaggedness, this gets back to one other thing I say in the book. I said some people in AI might not sign on to reverse engineering as a metaphor. Here’s something more of them might not sign on to, though I think most of them probably would, except maybe Richard Sutton, the coiner of the famous “bitter lesson” term and paper. It’s been most common to refer to pre-training and post-training as learning. I think that’s fair, but for some purposes it’s really more analogous to biological evolution. Language is a good example. During training, yes, the machines learn how to speak English, kind of, and that’s something a child learns, so that’s learning. On the other hand, they also develop this system for representing meaning, and, in the case of image recognition, these edge detection filters. Those, I’d argue, and I think it’s completely safe to say in the case of the edge detectors, and pretty safe to say in the case of a system for representing meaning, are products of biological evolution in humans.
The reason I bring this up is that there still prevails, in parts of the AI community, and, having listened to that pretty famous Dwarkesh interview with Richard Sutton, I think he’s an example of this, the idea that the human mind is a blank slate. That it’s like B. F. Skinner. I wish Dwarkesh had just asked Sutton, so you’re a hardcore Skinnerian? Because that’s the only way I could make sense of what he was saying. And I’d love to look further into the origin of the term artificial general intelligence. When I first heard it back in 2017, I thought, wait, do they think intelligence is just this general thing? Because, as I said, I’ve written about evolutionary psychology, and in that field the view pretty much is that evolution just kept improvising and building different things. So the mind is a jumble of functional things that, yes, are highly integrated, but it’s still not a general purpose intelligence machine. It’s good at threat detection, it’s good at sensing the emotional vibes of people, it’s good at making out visual objects, but these things evolved at different times, so there’s different equipment.
And I don’t see why we should expect the evolution of AI to be any different. So jaggedness is what I’d expect, because I don’t think there is a general blank-slatey thing. I was reading, or listening to, this biography of Demis Hassabis, and at some point he says, if we can solve intelligence, and I thought, that’s a weird phrase. I don’t think of intelligence as this one problem that you’ll at some point get the solution to. Maybe it’s out there, but I don’t think that’s what the human mind is. Look, who am I? With all due respect, the guy’s a genius, I’m not, and I don’t want to sound like too much of a jerk. But the answer to your question, if there are still these big gaping gaps in functionality, what’s your basis for hope, is: well, that’s the way evolution works. It’s in the nature of intelligence that it’s not just this general blank-slatey thing. That’s my view. Now, continual learning is a specific problem. First of all, if you’re talking about the practical impact, like can it replace all workers, it’s not hard for me to imagine simple shortcuts that should worry workers who want to hang on to their jobs. When Mark Zuckerberg starts monitoring the keystrokes of his workers, that’s one. You just update your LLM, or whatever, with fine-tuning, with the input derived from that monitoring. And as a practical matter you’ve solved the continual learning problem. You update it every once in a while. As a practical matter, that does not seem to me like it’s going to get in the way of the huge impact I’m talking about.
Inevitability, agency, and competitive environments
Henry: Can I ask about the attitude of inevitability that maybe I’m incorrectly reading into a lot of what you’re saying? It seems like there’s this path you see humanity as basically locked into now. I’m curious, firstly, whether I’m understanding that correctly, and secondly, how you’d respond to someone who says, look, sure, with evolution you’re dealing with certain deep grooves in some latent biological space. We’ve seen convergent evolution towards intelligence, multicellularity, social organisms. But technology’s different. We have more optionality with technology. Things could have been invented in a different order; if this country had gone to war with that country, things could have worked out very differently. So, do you see yourself as more on the technological determinist side, at least with regard to AI? And how do you respond to someone who says biology and technology are different, there’s more optionality in tech?
Bob: There is, in principle. In both cases, what we’ve seen so far, as a rule, is that what’s highly probable, if not inevitable, is the evolution of certain capabilities and properties. In evolution, it wasn’t inevitable, I think, that the first form of intelligence at our level would look like us, have five fingers. But the property was probably pretty likely all along. Certain things are independently developed many times in evolution: multicellularity, vision, flight. Those properties were likely to be discovered. I think you have the same thing in technology. But you’re right that in principle, because we are the environment of technology’s evolution, and unlike the environment of biological evolution, we’re aware that we’re playing that role, in principle we could change things.
There’s a guy at Google I still want to track down and invite on the podcast. He was on Rob Wiblin’s 80,000 Hours podcast a couple of years ago. He’d done what sounded like a PhD dissertation, and it spent a lot of time on the concept of agency: when can people exert true choice, true agency? As I remember the take-home lesson, it was when they are not in a competitive environment. If you’re a company trying to maximise stock value, you don’t really have much agency. Right now, if you’re Google or OpenAI or Anthropic, that’s why we have all these guys hinting, yeah, it might be a good thing for the world if we slowed down, but I’m afraid I can’t do that, because they’re in a competitive environment. That’s why I put so much emphasis on this, and I wish I’d put it in the book, I wish I’d tracked down this quote, because it’s a point of profound philosophical significance: true agency just cannot happen in certain kinds of competitive environments. And in the current international environment, humankind does not have agency over this technology. That’s the way to put it. So yes, in principle we do have agency in steering technology, and occasionally it’s been exercised, especially when it can be exercised at a national level. But sometimes it’s been exercised internationally, like ozone depletion, not to be confused with climate change. They actually handled that. It was a modest technological adjustment, more or less getting rid of aerosols. But it happened, and it happened internationally. So it’s possible.
Dan: Just one quick thing, Henry, then I’ll bring you in. I think the person you’re referring to is Allan Dafoe, on technological...
Henry: Yes, that’s what I was going to say. Allan.
Bob: Yeah, I think you’re right. God bless that man.
Dan: We’ll put a link to Allan Dafoe in the show notes. I think that’s a nice transition onto these questions about the idea of a pause.
The default trajectory and authoritarianism
Dan: As I understand it, a central argument in the book is that the default trajectory here is extremely worrying. Unless we get our act together, unless we achieve a certain kind of international cooperation and coordination, and we’re only going to do that if we overcome tribal cognitive biases, things are going to be really, really bad. So even though you’re not quite an AI doomer, because you think we’ve got a chance of intervening to avoid the worst-case scenarios, is it fair to say, Bob, that you think the default trajectory is extremely worrying and most likely really bad? And if that’s a fair characterisation, why do you think that?
Bob: If by default you mean the world failing to achieve an unprecedented degree of international coordination and governance, then I think it’s bad for more than one reason. One is just the magnitude of the earthquake. If this proceeds at its current velocity, completely unconstrained, with only the kind of regulation you can get at the national level in an environment of intense international enmity, which in many cases is not much regulation, then just the upheaval, when you add it all up, the social impact in America alone is going to be very destabilising. And you know what happens in periods of great disorder and chaos: it’s ripe for authoritarianism. As it happens, this technology is great for authoritarians. So this technology is both creating the circumstances for an authoritarian takeover and providing the tools.
This is a good segue to another part of the answer: geopolitical destabilisation. This is why I think Dario Amodei’s plan, unless he’s updated it, to literally get to superintelligence before China and bring China to its knees, and that’s a totally fair reading of what he says in “Machines of Loving Grace”, he doesn’t mention China there, but he says authoritarian bloc, and he’s co-authored things on China with one of the most extreme China hawks in America, Matt Pottinger, his plan is to bring China to its knees and then give it some ultimatum that he’s a little vague about. But my point for now is that getting to that point is so destabilising. The Superintelligence Strategy paper that Dan Hendrycks was lead author on, its most important point was kind of buried: if two nations both buy the premise that superintelligence confers hegemony, and buy the premise that, because of a dynamic of acceleration, which I think is evident, being a couple of months ahead in the race to this threshold could ultimately mean you attain complete hegemony, then obviously the superpower that’s two months behind has a strong incentive to resort to extreme measures. If they’re hearing from the other superpower what Dario Amodei in particular is saying, and what many American politicians are saying, which is we’ve got to win this race, and then America will, and this guy Alex Stamos, a very impressive, I’m sure good, guy, a famous cybersecurity expert, said America has to “dominate the 21st century.” That’s not quite as extreme as what Dario says, but if I’m China and I hear a bunch of Americans saying that, and you’re two months ahead, and it does look like things are accelerating, well, I might just resort to extreme measures. And it might not even stop with the most obvious thing, which is putting an end to the functionality of those factories in Taiwan, from which China is not getting any advanced AI chips anyway. It might go further than that. And you’re talking about two nuclear superpowers here. So you tell me, how much attention have you seen to the issue I just raised in the discourse? Maybe I’m missing something.
Dan: How much attention have I seen? I’ve seen a lot of debate over whether that strategy, which is being pushed by Amodei and others, is the right one.
Bob: But this specific point, that this invites attack from China? Again, it’s in the Superintelligence Strategy paper, I give them credit, I’d have flagged it a little more prominently. You’re starting to hear a little of it. Look, I don’t want to spend too much time complaining about what is and isn’t in the discourse. But this is part of the answer to your question, and I want to close it out by connecting it to the first thing I said. If we do what is basically Dario’s prescription, notwithstanding the occasional Anthropic paper that says, yeah, really, we should think about slowing down at some point, if we really pursue his prescription, which is in fact a breakneck race to superintelligence, because he genuinely believes this, and I think for him it’s not just a corporate talking point designed to fend off regulators, it’s ideological, and I genuinely respect that, he has principles, he has ideological principles, but I think they lead to a prescription that, if we follow it, not only courts the risk I just described but will lead to such rapid destabilisation of American society, which I submit, by virtue of its system of government, is less good at controlling instability than China is, that we may wind up, for reasons I described earlier, with authoritarianism coming to America through the back door. Dario’s whole thing is motivated by his fear of authoritarianism. He thinks China wants to make the whole world authoritarian. I haven’t seen evidence of that, but he thinks it’s the case. Fine. I’m just saying the kind of race he’s talking about has a good chance of bringing authoritarianism to America through the back door.
Steelmanning Amodei, and reading China
Dan: I think that’s a really strong argument, and one of the most compelling parts of the book. But I still think you have to look at the risks and dangers on both sides. You might steelman Amodei’s view by saying, look, of all the scenarios that could play out here, the current one, where frontier companies in the US are racing to superintelligence, all things considered is not that bad. For one reason: it’s happening in a liberal democracy. Saying that fifteen years ago would have been a bit more comfortable than it is now, given the Trump administration and so on, but relative to the political regime in China, these companies do exist in a liberal democracy, where there’s some kind of democratic accountability. And the frontier companies leading this race are demonstrating a surprising amount of concern with safety, genuinely employing AI safety and ethics researchers, putting a lot of funding behind that, being at least somewhat responsible relative to what you might expect from corporations like this. If you compare that scenario to one where, for example, China gets to superintelligence first, then it looks actually quite good. So if you could actually have an internationally coordinated pause, how confident are we that the scenario that would play out as a consequence would be superior to the current one?
Bob: Well, I’m not advocating that China get to superintelligence first. I’m advocating that we start right now having a very serious conversation with the rest of the world about nobody getting to superintelligence first. That said, if you accept the premise that superintelligence is this threshold that confers dominance, and goes beyond military dominance, which I don’t think is crazy, and I’ve never heard this spelled out, but probably somebody has, I assume they’re talking about things like being able to say to your AI, go infiltrate China’s social media, or America’s social media, and instigate a takeover of the government. I think they’re talking almost about that level of power. I don’t think that’s completely crazy. And I grant that if either nation gets that, I find it kind of creepy, honestly. Certainly if China gets it, I don’t feel comfortable that it would be a great world, which is why I’m not advocating it.
But I also think there’s an assumption that right now part of China’s foreign policy is the aspiration to remake the world in its image, that it wants other nations to have its system of government. I think that premise is almost wholly without supporting evidence, because it’s another one of these things that just doesn’t get debated much in the mainstream halls of Western discourse. I think China is, in a way, the opposite: a very pragmatic, realist country. America, obviously, has said it is our aspiration, it’s Dario’s aspiration, to remake the world in our image, we want liberal democracies everywhere, and if you’re not one now, we will sanction you or invade you or bomb you. That is our policy. And, by the way, is a country that just seems to have a habit of invading and bombing other countries the country you want to get to superintelligence first? If you’re an observer from Mars looking down, China hasn’t attacked another country since 1979, and America has done it under every single president it’s had since 1979. Is it so obvious to an objective observer that things will work out fine if America has the superintelligence? Maybe you could say, well, then they won’t have to invade countries. I just think the premises underlying the whole discourse about China are not as carefully examined as I’d like.
But I mainly want to emphasise: I’m not saying let’s let China get to superintelligence first. I’m saying job one is just to see if we can talk about slowing down a little. And granted, that would have to be verifiable. That’s why my book’s proposals in the non-political realm seem to some people hopelessly ambitious: encouraging people, including in America, to get a little better at transcending some of the cognitive biases that undergird the psychology of tribalism, so that it will be less easy for leaders who want to inflate our fear of other countries, and get us to bomb countries like Iran, to do that. I know that’s ambitious, and I’m not saying it’s probably going to work, it probably won’t work, we’re probably toast. But my argument, and the book is an argument, is that we have to, as a species, get a little closer to enlightenment. I’m not just talking about America. In all these countries, all populations are pathetically susceptible to threat inflation, and it’s always in the interest of political leaders, and others like arms makers, to inflate threats. So I want to emphasise what I’m saying: not let’s let China take over the world, but how about we understand that it’s in our mutual interest for nobody to do that, and the first step would be to have rich enough discourse for some degree of slowing down and international coordination to not sound completely crazy.
Henry: There’s a lot I agree with there, but two areas I might push back on. One: yes, China hasn’t had a recent track record of going to war with its neighbours, but it has absolutely acted in quite aggressive ways towards a whole bunch of countries in the West Philippine Sea. It’s engaged in constant low-level aggression with Vietnam, the Philippines. It’s been building artificial islands and reefs. It’s been on a large-scale international enterprise of loans via the Belt and Road Initiative, giving it more and more control over countries around the world. Okay, it’s not bombing Iran, but there are lots of very reasonable, not crazy war-hawk, people who look at the way China’s acting and say this is a country that does have a global agenda that we should be aware of, and that is at least to some extent in tension with that of the United States.
Maybe the second, more fundamental point I’d press you on: the worry here is not necessarily that China’s going to get superintelligence first, but that superintelligence is really one of the very few arrows in America’s quiver where it currently has a significant lead over China. On a whole bunch of other really sensitive military or dual-use technologies, from drones to robotics to shipbuilding to batteries to energy generation, China’s steaming ahead of the West. Superintelligence may be the only really decisive lever the West has if it doesn’t want to end up with a China-dominated century. So what would you respond to that?
Bob: Quickly on the last point before I return to it: I’m not advocating that we have a century dominated by any one country. I’m advocating that we avoid that. On the first point, I assume you’re not saying that the things you mentioned, like the Belt and Road Initiative, throwing its weight around regionally, distinguish it from America. You’re not saying that, right?
Henry: No, I’m not saying that. But equally, there’s a certain tendency to paint China as fundamentally an inward-looking power that only wants to secure its own economic future, and I don’t buy that. China does absolutely have a global agenda that involves increasing its influence in Africa, in Latin America, in absolutely dominating the South China Sea, or West Philippine Sea, pressing land claims and so forth. So it’s not this purely introverted power that some people make it out to be. I’m not suggesting you’re suggesting that.
Bob: On the point that it’s not just concerned about its national security: I had on my podcast Dmitri Alperovitch, who founded CrowdStrike and had written a book, and he’s a China hawk. I said to him, I read your book, and the great thing about it, as China hawk books go, is that when I look at your assessment of China’s motivations and why they want to exert this regional influence, it can all be explained as a means of preserving the trade conduits it depends on. He emphasises that, well, there are certain parts of the ocean that are super shallow, and I forget how it all works out. But the main thing is, and people can check the podcast to make sure I’m not exaggerating, he said, well, yeah, that’s true. And I want to say: the biggest human cognitive contributor to war, related to threat inflation, is the tendency of people on both sides to view things that may be defensively motivated on the other side as offensive. Political scientists have a term for this; the term security dilemma kind of encompasses that. I’d say it’s not exactly that, but it encompasses it as one of its main strands. You see it again and again. You see it in the way Americans think of China, the way China thinks of Americans, the way Iranians and Israelis think about each other. I can go on all day citing chapter and verse. Famously, this may be the reason World War One happened. It’s a positive feedback cycle: they did this thing, that’s a threat to us, so we’re going to do this thing, and then the other side sees that as a threat, and so on. It’s been much commented on.
I just wanted to say that what you said, that China throwing its weight around regionally is a sign of aggressive aspirations that are not purely defensive, not purely in the name of national security, I’d say that’s an example of that dynamic. But before I go on, I can pause while you rebut me.
Henry: I think it gets a bit blurry, and I agree that very often seemingly aggressive measures are motivated by insecurity. That’s basically the history of a large part of the Cold War: the whole idea of containment, worries about domino strategies. So there’s no neat distinction between defensive versus aggressive powers. But the one area I’d say is that, given that the United States and China do constitute very different ideological visions for the future, I don’t think it’s the case that totally peaceful coexistence is the default option. I think they have quite distinct interests in different parts of the world, whether that’s the United States and the Monroe Doctrine in Latin America, which is obviously at odds with China’s vision of having large amounts of influence, economic mainly, but also perhaps political. But I’m not sure that’s a fundamental disagreement, and I do want to hear what you say about my other point, which is that superintelligence is one of the very few assets the United States has if there is going to be this competition.
Bob: Again, my premise is that we have to make sure there isn’t the competition. And it’s related: you’re asking what the chances of peaceful coexistence are. I’m not saying they’re high. I’m saying, and it’s an argument, so it could be wrong, but it’s an argument laid out in the book, that this technology has properties that force us to fundamentally rethink things we’ve always taken for granted, like that the peaceful coexistence of all the world’s nations is impossible. There was a time when a lot of things we take for granted were considered impossible. If you’d said seventy, eighty, ninety years ago, sixty maybe, right after World War Two, that someday France and Germany would have the same currency, the reply would have been, well, which country will have conquered which? You wouldn’t have imagined that. As it turns out, the single currency has had its upsides and downsides. I’m just saying that forms of cooperation and collaboration that seem inconceivable do happen. There was a time when there had never been institutionalised peace among groups of people more than ten miles apart, and now we have huge swathes of peaceful coexistence. There’s Peter Singer’s book The Expanding Circle, which argues that there’s been a kind of moral advance over time that humankind has exhibited. Peter and I have somewhat different ideas about what the main engine of that has been, but I think it’s happened. So, as I understand it, your question is, if we don’t dominate China with superintelligence, how will we dominate China?
Henry: Or rather, the worry is that if the United States does not use superintelligence to at least equalise the power differential with China heading into the 21st century, then China’s massive and growing lead in other strategic technologies, drones, energy production, shipbuilding, will mean the 21st century ends up being one dominated by the ideology of the CCP. That’s the worry.
Bob: Well, most people who talk about superintelligence and assess its strategic significance don’t take it as the kind of thing that could balance China. It’s like, balance their drones? No, if we have superintelligence, their drones won’t matter, we’ll have better drones. It’s a technology of global dominance, according to most people who think about it. So to think of it as something that will just maintain the balance, I don’t, now, I haven’t thought this through, I’m a little agnostic about superintelligence, its properties, its likelihood, its date of advent, but the premises underlying most of the discussion of it, as I understand it, preclude the possibility of one nation having it and it being a balancing thing. No, it’s a dominating thing, in these discourses.
Henry: I’m inclined to agree with that, and then I want to pass back to Dan to move the show on. But I think it illustrates how a lot of one’s commitments about the nature of AI hang together. If you really think AI superintelligence is just a transcendental technology in some sense, then the idea that it could ever serve as a geopolitical balance doesn’t make any sense: whoever controls it determines the future of our light cone, let alone the future of the 21st century. On the other hand, if you have a more “AI as normal technology” view, then superintelligence might start to look like something that could be in column A to counterbalance other industrial capabilities in column B. So it’s funny how a lot of these different parts of the AI picture hang together.
Bob: Yeah. And actually it leads to a question for Dan. Would it still be a normal technology if you had superintelligence? Is superintelligence part of the normal technology paradigm?
Dan: No. It’s complicated. The way they frame things is they contrast the normal technology worldview with the AI-as-potentially-superintelligent-species worldview. But it’s complicated, because Arvind Narayanan and Sayash Kapoor, the authors behind “AI as Normal Technology”, grant that AI systems could in principle be much more capable than human beings along many dimensions. It’s just that with certain capabilities, like persuading people against their self-interest, and also forecasting, they seem to be quite sceptical. But then there’s this other very strange feature of their worldview, where they think that even if you do think really transformative AGI is possible, we’re not in an epistemic position to forecast what would happen in that world anyway, so we shouldn’t really talk about it. Which makes the whole worldview quite difficult to summarise and engage with. It’s a little slippery, and it bundles lots of different things together.
Negativity bias and the discourse
Dan: Maybe I could talk about one other cognitive bias, because there are a couple of things I really want to get to that have to do with the appendix. You’re really good on tribalism and how it can distort our judgment and perception in counterproductive ways. There’s another bias, negativity bias, and the way it interacts with the incentives of the media and the pundit and public intellectual class. It interacts with tribalism in various ways, but I think it’s one of the most consequential and damaging biases in the modern world. In liberal democracies today you’ve got one of the most peaceful, prosperous societies in human history, and yet large segments of the population basically think they live in a dystopian hellscape, which is why they want to vote for anti-establishment politicians. A lot of the reason for that is the way our evolved tendency to attend to negative and threatening stimuli interacts with the incentives of a very competitive media environment that is constantly broadcasting negativity and threat and outrage. And I worry that this is also happening when it comes to AI.
My view is that at the moment, and we can talk about scenarios and how they’ll play out in future, AI is overwhelmingly a positive technology. I think it’s having very beneficial consequences for the information environment. I think it has really beneficial consequences even when it’s used for things like counselling, to help people with their mental health, and it’s enhancing productivity. And yet when you look at the discourse about AI, it’s so negative, often in quite misinformed ways. Think about the backlash to data centres and all the misinformation about water use. Think about, and this is an area I’ve got interest in and you write about in the book, the doom and gloom about the impact of AI on the information environment, which I don’t think really tracks the reality of how these systems are affecting how people form beliefs and gather information. And I’m a little worried that maybe in the book you might also be skewing too negative, too much in the direction of catastrophe, relative to what the actual reality is. So what do you think about that worry, that we’re too negative, too alarmist, too catastrophising about the technology, relative to what we actually observe?
Bob: So you’re worried that I worry too much?
Dan: Potentially.
Bob: Fair worry. I agree the technology has tons of positive things. I use it, it’s enthralling. That’s one reason I think the current air of opposition to it, at least in America, is a very fluid thing. I don’t know how it’s going to play out. I think a lot of the people who are opposed to it haven’t discovered positive uses they’d discover if they used it more. And all the things it could do, cure cancer and so on, would be great. I actually used it, I had cancer a year ago and I’m fine now, and it came in handy in particular ways, more as a research tool. I don’t doubt it can do a lot of the miraculous stuff.
Now, as for a negativity bias, first of all, as a cognitive bias, I’ve always thought it couldn’t be a quite systematic bias, because I think natural selection designed us to be unduly optimistic in some circumstances. For example, heterosexual males see a woman in the distance and overestimate her attractiveness relative to what they’ll eventually judge when they see her close up. That’s a kind of positivity bias. And there are probably circumstances in which they’re inclined, on average, to overestimate how attractive they’re being found by other people. Consequentially, I think there are certain group dynamics of optimism that are natural and unfounded, and you see that on both sides in the run-ups to war: yeah, we can win this thing. That may have some basis, I’m less sure in evolution, but it’s a thing, you see it all the time. But yes, are people also inclined, quite a bit, to worry about things? Yes, you’d expect natural selection to create animals attentive to threats. Watch a squirrel while it’s eating its nuts, always looking around for stuff to worry about.
As for whether we need more of it or less of it, I don’t know. I can certainly cite examples where I was worried and history bore me out. I was opposed to the Iraq War, and most people in America say, yeah, that was a mistake. I was opposed to the recent attack on Iran; that didn’t work out. America has had so many foreign policy misadventures in my life that were preceded by what in retrospect was undue positivity of a kind. And you can come up with specific examples, like thalidomide, you’re too young, but thalidomide was a drug they gave to pregnant women without enough contemplation. I knew a guy in college who was a thalidomide baby: he was missing the part of his arm between his elbows and his shoulders, his forearms came directly out of his shoulders. So you can come up with examples on either side. I’d just say, if you’re going to have as much positivity coming out of Silicon Valley about technology, and it’s going to be as well funded as it is, you definitely need people like me, just so you can have a debate. You’d agree debate about this stuff is good. You need both kinds of people: optimists to argue with me, and pessimists to argue with them. I don’t know whether you can point to cases where the balance works too much in one direction or the other. That probably wasn’t a satisfactory answer.
Dan: No, it was. I completely agree that sometimes worrying is completely legitimate, and when it comes to AI specifically there are things to worry about. I just think in general there’s a systematic mismatch between how negative people’s picture of the world is and the actual empirical reality. There’s lots of data on this, it’s the kind of thing people like Steven Pinker bang on about. People think crime is increasing; actually violent crime has dramatically decreased. People think back in the good old days we used to be so wealthy; actually we were a lot poorer in the past than we are now, and so on.
Bob: And people think China wants to take over the world. So I’m kind of on the same page, now that I think about it. I’m just saying there are cases where I’d argue the bias is too negative, like China, and cases where other people would argue the other way. I don’t think the world faces some kind of uniform negativity bias. I think people of certain ideological persuasions, very pro-free-market, see a certain kind of negativity, and people of my foreign-policy persuasion see another kind, like threat inflation. And you just need to argue about stuff. As evidence that I don’t think it’s a uniform bias against technology: the social media revolution was not accompanied by this degree of negativity. Not nearly. People were pretty upbeat about it. And the way that played out, and Dan, I know you’re sceptical of the claim that social media had a polarising effect, but for present purposes the relevant point is just that a lot of people think it did, or think it messed up their kids, and they feel like they were burned by Silicon Valley optimism. That’s one thing going on now with AI that informs the reaction against it. But my main point is, I’m older than you, and I’ve been through periods of tremendous technological optimism. It’s not a uniform human thing. So maybe we should pay attention to it when it happens.
Dan: Yeah. The social media thing opens up a whole other can of worms, and we should be time-conscious, so we can’t get into it now.
Does evolution have a purpose?
Dan: I really wanted to end by touching on the appendix of the book, where you have these spicy takes that are likely to interest us as philosophers. I should say the majority of the book...
Bob: I like that word, because otherwise I doubt anyone will read it. Spicy. That’s as sexy as you can make this subject sound. Thank you.
Dan: The bulk of the book is a down-to-earth, concrete analysis of the technology and the political economy. But it’s suffused with something more, and it really gets to it at the end. You’ve got an argument about whether the evolutionary process has a purpose, is directed in some sense, and an argument about consciousness, which I found really interesting. The first argument, about evolution itself maybe having a kind of overarching telos or purpose, do you want to walk us through the basic argument?
Bob: Yeah, first, to emphasise, that doesn’t mean you’re departing from a strict materialist view of the way either biological or subsequent cultural and technological evolution unfolds. Machines can have purposes; material processes can have purposes. And a purpose, a telos, can be imparted in a lot of ways. It can have an intelligent designer in the conventional sense, the way we design machines that have a purpose. But it can also be imparted by what you could maybe call intelligence. There’s an argument that natural selection, for example, well, let’s not go there. Let me just say you could argue that natural selection imparts purpose, or a goal, to organisms, namely to get genes into the next generation. That is the criterion of “design,” and I put it in quotes because natural selection isn’t an intelligent process, isn’t an intelligence in any conventional sense. It’s an information processing system, but it is not, we assume, some kind of conscious designing intelligence, and in any event nothing like the ones we know. People like the late Dan Dennett didn’t think evolution has a purpose. And by the way, I’m strictly speaking agnostic. I don’t know. I do point to things that I think add weight to the case that the unfolding of evolution has some purpose.
What I just said about the sense in which natural selection imparts purposes to organisms means that even if evolution has a purpose, that could have been imparted by a process. I trot out this variant of Lee Smolin’s cosmological natural selection, which involves a replication of universes, according to which, if intelligence in a universe facilitates the reproduction of the universe, then you can imagine Lee’s dynamic leading to it. I’ve discussed this with him on the podcast, and I was happy to find that a couple of people have actually posited that a version of his selective replication of universes scenario, cosmological natural selection, would favour universes that give birth to systems that lead to intelligence. So in this scenario, if you want to play it out, Lee says, well, maybe black holes are portals of replication, in which case that would explain why there are so many black holes, because this process would favour universes with black holes, since those universes are prolific. Well, it could be that superintelligence helps us make more black holes. This is a wild example, of course. So my point is, if evolution has a purpose, it could be an old-fashioned deistic god, could be aliens from another planet, could be something that’s not intelligence in either of those senses.
That said, I try to draw a kind of ironic support from Dawkins’s book The Blind Watchmaker, to a very limited extent. Of course, he didn’t think evolution has a purpose, but his mode of analysis there does at least acknowledge that you can inspect systems for signs of purpose.
Henry: Just to get really clear: there’s absolutely a notion of purpose you can see in biological evolution that Dennett’s really happy with. He uses the kinder vocabulary to describe how evolution kind of aims towards certain goals, and using techniques like dynamical systems theory, talking about state spaces and equilibria, you can say evolution is going to tend towards certain kinds of highly adaptive designs that fit their environments, given certain environmental constraints: multicellularity, flight, vision, and so forth. But it sounds like there’s something a little more heavyweight in your notion of purpose.
Bob: Yeah, well, these are different levels of organisation. Dan wouldn’t say evolution has a purpose. He’d say evolution instils purpose in organisms. The question is, if you move to a higher level of organisation, look, if you step back and look at the three and a half billion years of life, you do see some directionality. You move to higher levels of organisation, richer complexity: multicelled life, societies of multicelled life. One society of multicelled life, ours, launches a second kind of evolution that carries organisation to higher levels: hunter-gatherer village, ancient state, and so on. Now we’re on the verge of globalisation. We’ve built something that looks kind of like a global brain, just with the internet. And, as I note, people were talking about a global brain a hundred years ago; it was discernibly developing a hundred years ago. So it has some things in common with the maturation of an organism, if you step back far enough. That’s what an organism does: it starts out as one cell, and then you get functional differentiation and all these things that I’d argue you see in the course of evolution. I had a whole conversation with Dan Dennett about this, and it’s on YouTube. But I want to be clear: Dan agreed, and he died three or four years ago, that evolution, you might want to put design and purpose in quotes, fine, “designs” organisms, “imparts” a purpose, just getting genes into the next generation. But he wouldn’t say evolution’s purpose is to get genes into the next generation; it’s the purpose instilled by evolution in the animals. And I’m stepping back and saying, well, what if the whole process has a purpose? What if the first life was a seed for the evolution of an organism? I’d say you step back and look at the process, and there’s more evidence than there would have been if nothing particular had happened after the first self-replicating strand of information occurred on this planet and you never got complex cells. Even if multicellularity had evolved only once, but it evolved many times, and that suggests it was likely to happen. So there is evidence you can adduce. But I just want to be clear: the question I’m addressing in the appendix, and I’m not claiming to be confident of an answer, is whether purpose resides at that higher level.
Dan: Sorry to cut you off, I was just going to say, I think it’s a very interesting argument. Where the analogy breaks down a little for me is: if I come across a watch, it shows clear evidence of design, so I want to understand the design process by which it came about. If I come across an organism, again it shows clear evidence of something that looks like design, so we appeal to something like evolution by natural selection that could give rise to that appearance. But if you’re looking at the whole evolution of life on the planet, I agree you’re encountering a lot of complexity, but it’s not clear to me why the entire system itself would scream design of a sort that couldn’t be explained just by the nuts and bolts of evolution by natural selection.
Bob: Okay, let me ask you: what if, instead of looking at a mature organism, we observe an organism throughout its ontogeny, its entire maturation from single cell to organism? You view the way the cells replicate, they differentiate, and you understand the mechanism, and you see that this was all likely. It’s not like, if you took this same cell and put it in the same circumstance, this whole unfolding towards a mature organism would be unlikely, a fluke. And you wind up with this big brain, say it’s a primate, that governs this whole body. You carry the functionality to higher and higher levels, and then you get this coordination of the whole thing via a brain, and you understand that the process was driven by things that made it very likely to happen that way. That’s why, if you’re going to argue about evolution, it does matter that multicellularity evolved many times. Now, people argue that there have been certain thresholds that were unlikely to be crossed. That’s an important part of the argument. There was a book by John Maynard Smith and another author called The Major Transitions in Evolution, and that was their view: there were things that were unlikely to be crossed. Fine. But this argument about how likely we were to get to intelligence, some great biologists agreed with me on it, including William Hamilton, author of kin selection theory. He agreed, and further agreed that it did kind of suggest that maybe aliens planted the life. To him it seemed so likely that one species or another would get to intelligence that maybe it did have a purpose. So he bought not just the premise that the unfolding towards some kind of intelligence had been likely, but that it was some kind of evidence of purpose. That too is on video; it’s in a New York Times piece of mine, the video snippet is called “Does Evolution Have a Purpose,” I think, so you can watch him say it. And my question to you, Dan, is: is that a legitimate part of the consideration of whether the organism has a purpose, the seeming directionality of the unfolding towards this brain that coordinates the whole thing?
Dan: Where I stand, I don’t find it a crazy idea, and I found the argument interesting. But to me there seems to be a big difference. If we think about an organism and that process of development and maturation, it’s culminating in a kind of unified, bounded, purposive system that can survive and reproduce effectively in its environment. Whereas when I think about the evolution of all life on earth, it doesn’t seem to be culminating in a really unified, bounded, coordinated system in the way you find with an individual organism, at least as of now. I mean, it’s part of your argument that you think that’s the direction, that we’re looking like we’re going to culminate in something that looks a lot more like an individual organism. Is that the argument?
Bob: Yeah. I’d say already you could say the global economy is like a global brain; it’s an information processing system that allocates goods and services. But the governance part, the literal coordination, what do brains do? They coordinate bodies. I’m saying that part of it, there’s a part in the final chapter where I say it’s almost like what our species is being told is: look, we can do this global brain thing the easy way, or the hard way. The easy way, compatible with your ongoing welfare, is for you to design a system of international governance that minimises the concentration of power, no more than is necessary, that is democratic and decentralised, and in which you continue to flourish. Or things are going to get ugly. At least in the doomer scenarios, and in the book I look at the escape scenarios, where AI escapes our control or takes over, and find that they’re not crazy, which I thought they were ten years ago. Or maybe something like that happens, or maybe just a human authoritarian takeover by somebody empowered with AI. But I do think that if we do not understand how imperative it is to start coordinating internationally, and to do it in a way I’d like, it may get done in a much uglier way, involving more death and suffering and ultimately less human freedom in one sense or another. So my answer to your challenge is: the story’s not over. I think it’s actually pretty likely that you’ll wind up with some global coordinating mechanism. I want it to be the good kind. But the jury’s out.
I will say, of course, that when you talk about purpose in evolution, you’re working at a handicap, because we only have one data point. When we look at animals, there are tons of members of each species, and tons of species, so it’s much easier to conclude that something gave them the purpose: they all do this one thing, get their genes into the next generation, there must be some reason for that, and that’s what Dawkins would agree with, that part of the analysis. But we only have one instance of the whole process, and that’s why I say, if you’re interested in this question, and a lot of people aren’t, and that’s fine, then you have to inspect the ontogeny, the dynamics of the unfolding and its directionality. That’s the only data you have to work with.
Consciousness and epiphenomenalism
Bob: And then, as you know, I have this very speculative argument about consciousness in the appendix. If we’re out of time I’ll just say, for philosophers, people interested in the mind-body problem: I contend that what we call epiphenomenal consciousness, which could be the kind of consciousness there is, could have a function, if that function were imparted at the higher level of organisation by the designer of evolution, whether that’s a system of intelligence or whatever. But an epiphenomenal consciousness could not be something invented by natural selection on this planet, unless you think it’s of very recent origin. If you think prehuman animals have consciousness and it’s epiphenomenal, it could not possibly have been designed by evolution for a purpose, because it doesn’t start affecting reality until you get animals with language that talk about it, at which point in some sense it ceases to be epiphenomenal. This conversation we’re having is a function of the existence of consciousness, even if consciousness was epiphenomenal for the first however many billion years.
Henry: Just on a point of clarification. When people use the term epiphenomenal, at least in regard to consciousness, there are two main ways they can use it. One is that it’s an evolutionary epiphenomenon, a spandrel, in the biological terminology; it doesn’t have an evolutionary function itself. The other is the more metaphysically laden notion, where it is metaphysically or ontologically epiphenomenal: it’s not within the inventory of the physical, it’s a non-physical phenomenon that supervenes on the physical world. Am I right in thinking you mean the first one, or something in between the two? Some secret third thing?
Bob: Well, I wouldn’t call it, I don’t think, to be an epiphenomenalist, as I understand it, you have to think consciousness itself is physical. You think it is affected by the physical body, by physical information processing in the body, but does not affect that. Again, until we start talking about it, and not just talking about it academically, but saying, I feel bad, you hurt me, that hurt me. Those things change, and that enters into my argument about the sense in which it could be functional at the social level, an integrating force in a sense, but that function would have to have been imparted at a higher level of organisation. There’s another reason I think that, which is that I think of consciousness as an inherently deeply metaphysical thing. It would have to be a property of the universe. The idea that natural selection invented it in the sense that it invented toes is, to me, just crazy. Anyone who says that does not share my understanding of what consciousness is, which of course many people don’t, but I just think consciousness has these completely unique properties. It’s private; no other person can observe it. There’s nothing else in the universe we know of that’s like that. So that’s weird and profound and metaphysical, and I just don’t imagine natural selection producing a mutation that does that. The mutation would have to be at the level of the universe, I think. Like, this is a property of information processing in this universe, kind of.
Henry: So it sounds like the picture you’re broadly sympathetic to is a kind of David Chalmers-style picture, where consciousness is non-identical to any physical property but supervenes on physical properties, thanks to something like brute psychophysical laws that just obtain within this universe: when you get the right kind of collocation of physical structures, you get this further thing that is not itself physical, which is consciousness. Is that a rough reconstruction of the view?
Bob: I think that’s fair. On epiphenomenalism, I actually had a conversation with him on my podcast, and I think we have a slightly different view on how we have to define epiphenomenalism, posed by the fact that it seems to start affecting the world as soon as animals with language start saying things about it. There was a metaphor I didn’t have at my disposal when I was talking to him that I’ll use now. When I talk about a consciousness that was epiphenomenal until people start talking about it, he has trouble with that happening, and I think has to define it away via his definition of epiphenomenalism, or something. No, that’s probably not fair, so dismiss whatever I just said. But a good metaphor for epiphenomenal consciousness, to begin with, is a shadow. My hand moves along, it has a shadow. The movement of my hand affects the shadow; the shadow doesn’t affect the hand. But once I have eyesight and look down and see my shadow, the shadow can affect my behaviour. I think the advent of language, with respect to an epiphenomenal consciousness, is like the advent of eyesight with respect to a shadow: it imparts causal import to a previously epiphenomenal thing. I should say, I’m not a dyed-in-the-wool epiphenomenalist on consciousness. I’m a mysterian; I don’t know what to make of it. Epiphenomenalism is the most intuitively appealing thing, but since that raises the question of what it’s for, my appendix proposes an answer. There’s something it could be for, but only if the functionality was imparted by something that created evolution, in the scenario where evolution has a purpose.
Dan: I’m conscious of the time, but maybe I’ll throw in one quick, potentially clarificatory question, potentially objection, just to get a better sense of your view, and then we can wrap up. I suspect we’d really have to do a whole conversation on this to get into the weeds, but we’ll get you back on. My thought was something like this. You’re an epiphenomenalist...
Bob: I’m available. Not today, but I’m available.
Dan: ...it seems in part because you think you can explain behaviour in general, but you find it at least somewhat plausible...
Bob: First of all, I’m not an epiphenomenalist. It’s a plausible, it’s the most intuitively appealing view of consciousness to me, but I have no idea what consciousness is.
Dan: Fair enough. Okay, but you find it plausible because you think, in general, setting aside the fact that we talk about it, we can explain our behaviour, and neuroscientists often do explain our behaviour, without appealing to consciousness, which is this weird thing. Firstly, I don’t quite agree with that, because I find things like the global workspace theory of consciousness quite plausible. But even setting that aside, if you think that about our behaviour in general, presumably it would also apply to our verbal behaviour. So why wouldn’t you also find epiphenomenalism plausible in the case of us talking about consciousness?
Bob: You mean it facilitates the use of language, or, wait, maybe I missed something.
Dan: Well, the thought would be, as I understand your argument, it’s...
Bob: Why couldn’t it still be epiphenomenal?
Dan: Presumably the same argument would go through. I’m making these sounds...
Bob: As I understand it, it’s part of the definition of epiphenomenal that the thing you’re calling epiphenomenal does not affect the material world. And right now the particular physical sound waves we are exchanging would not be in the form they’re in if consciousness did not exist. If subjective experience did not exist, we would not be having this conversation. Most conversations people have, you can imagine happening even if there were no sentience. You can imagine complex, smart organisms that communicate in as sophisticated a way as we communicate without it being like anything to be them. But you cannot imagine this conversation, us talking about what it’s like to be us, if there were no such thing. And you cannot imagine somebody saying, ouch, you just stepped on my toe and that hurt. You can imagine them saying you stepped on my toe and that threatens to damage my toe, but you can’t imagine explicit references to internal states if they don’t exist. That’s what I’m saying. This problem is acknowledged, and I forget David Chalmers’s workaround, but it’s in recognition of this challenge to epiphenomenalism that he does some complicated thing that, as I recall, kind of defines the problem away. But the straightforward way to describe it is: if consciousness is epiphenomenal, it ceases to be, in the strict sense of not having any effect on the physical world, once you have animals that communicate about sentience, about feelings and thoughts. That’s the argument. Does that make sense or not?
Dan: It makes sense. I think we’re going to have to have you back on for a fuller conversation about this.
Bob: Well, I just said everything I know about it. But yes, I’m game for coming on again in principle, for sure.
Dan: Fantastic. We really appreciate you coming on. I highly recommend the book. I think it’s one of the best books on this subject, one that people will get a lot out of even if they’re already immersed in the literature. Henry and I will be back in a couple of weeks. So thank you, everyone.
Henry: Thanks, everyone, and thanks, Bob.
Bob: Thank you. This was a great conversation. I really appreciate it.
