It is now almost ten years since the EU referendum. There will, no doubt, be an article in every newspaper next month detailing what Brexit has meant for the economy, national sovereignty, migration patterns, fishermen, farmers, and so on. But for me, by far the biggest change that the referendum brought about was the creation of two new political tribes: Remainers and Leavers.

Over the last decade, not only have more people in Britain claimed a Brexit identity than a party identity, but people’s emotional attachment to their Brexit tribe was, and is, substantially stronger than their party attachment. Membership of these new political teams, created over a few months, is more important to people than the party identities that dominated British society for the last century.

At first glance, this might seem strange. Before 2016, most of us had very little interest in the EU. When David Cameron said that he would call a referendum on membership in January 2013, only 2 per cent of people said that the EU was the most important issue facing the country. The referendum thus forced people to make a binary choice on an issue about which they did not have very strong feelings.

Before we vote on something, we can have ambiguous, changeable attitudes, but after voting, we resolve that ambiguity by choosing one side or the other and committing ourselves to a named group of fellow travellers. The fact that this tribal loyalty was then tested over years of wrangling over the actual outcome of Brexit (in 2018 and 2019, MPs said the word Brexit in their parliamentary speeches every five minutes on average) meant that people had the opportunity to rehearse and reinforce their new identity again and again.

Subscribe now

Why are these new political tribes interesting? I think there are three reasons, and the first is that we got to see a rare event: new political identities being born. Before 2016 nobody thought of themselves as a Remainer or Leaver. Although characteristics like education, age and national identity were predictors of people’s attitudes to the EU issue, and ultimately their referendum vote, we absolutely cannot reduce the two sides to simple caricatures based on class, age, education, or even national identity.

As we show in the book, the vote was not simply an exercise in counting up existing groups who were pro-EU or anti-EU. Rather, many people who had similar middling views about the EU were forced by the referendum to make a choice in 2016 and plump for one tribe or the other. The decision people made on 23^rd June then became a part of how they saw themselves and how they wanted others to perceive them.

Second, despite their overnight creation, these new political identities proved remarkably resilient and strongly held. By 2017, there was a small cottage industry in articles about how to avoid Christmas family rows about Brexit. Relationship counsellors, psychotherapists, and even hostage negotiators were asked by journalists how to defuse clashes between the Brexit tribes. Why was this seen to be necessary? Because new group identities meant new emotionally resonant in-group loyalties and out-group hostilities.

To better understand this, we use survey questions that focus on the degree to which people naturally identify with their group. For example, we asked people whether they usually said ‘we’ instead of ‘they’ when they talked about their own Brexit tribe. When the football team you support loses, you say ‘we played badly’, even though you never set foot on the pitch yourself. It is the same idea here.

Combining many measures like that, we find that Remainers and Leavers were consistently a lot more attached to their identity than were Conservative or Labour supporters. And those scores have been very stable over the last ten years. People like people like them. And they define ‘like them’ in terms of their Brexit tribe. All our measures also show that people not only disagree with, but really dislike, people on the other side and typically say that they have a ‘cold or unfavourable feeling’ towards their rival group. Again, this has barely changed since 2016.

Third, people engage in the same sort of motivated reasoning that we see for party identities. At the most basic level, any group identity that is strongly held will provide motivations to think that the other side is inferior and should be avoided. As our data shows, huge majorities say that their own Brexit group is intelligent, honest and selfless, while the other side is stupid, dishonest and selfish. In fact, when we asked people to describe the other side in their own words, a quarter simply listed bad things and another quarter did that in addition to other information (to give you a flavour, one of the pithiest responses was simply ‘selfish dicks’). However we measure it, we find widespread prejudice. And we also find lots of evidence of discrimination: people actively wanted to avoid everyday interactions with people on the rival team.

It is tempting to think of Americans as peculiarly politically divided, but the levels of hostility, prejudice and discrimination between the Brexit tribes are all as large as, or larger than, any partisan differences in the US. And if you have been reading this smugly thinking that this is just true of those foolish people on the other side, then think again, because almost all the consequences of tribalism that we reveal in the book are symmetrical: Remainers and Leavers are just two sides of the same coin.

For me, the aspect of motivated reasoning that is most interesting is how it shapes perceptions of the state of the world and remedies for its woes. For party identities, we normally think about politicians providing stories for people who identify with their party to tell each other. For the Brexit tribes, this is much less of an option, since there are no formal group leaders. And yet people were, and are, quite capable of independently searching for, and believing in, messages that support their own side’s view of reality, and then ignoring or rationalizing away information that contradicts that view.

Interestingly, sometimes that means not bothering to shop at the ‘marketplace of rationalizations’ at all. The difference between Leavers and Remainers over whether they thought that the outcome of Brexit on Britain would be positive or negative is enormous: nearly 3 points on a five-point scale. That has barely changed in ten years. Yet when we asked people, ‘what are those positive or negative effects?’, well over half of both Remainers and Leavers were unable to actually name anything specific. In short, if my side voted for the change, I say ‘good’; if my side voted against the change, I say ‘bad’.

This suggests that it may be the signalling aspect of motivated reasoning that dominates under these conditions. In other words, our Brexit identity influences our political opinions because we want to display the fashions of our group. But as there are no party leaders telling us what to believe, no fashion icons telling us what to wear, this process depends on knowing what other people in our tribe think. This limits our ability to change our policy opinions to match our tribe. On one issue we do have a very strong sense of what both sides think: Remainers love the idea of European integration and Leavers hate it. As we show, initial large differences in attitudes towards the EU became even larger after the referendum, as people sought to become good group members and adopt their group’s norms. But this also applied to some other policy areas, like immigration, about which people knew, or at least thought that they knew, the group norm.

There is a final key area in which we see both sides rationalizing away information that is inconvenient. It is always true that people who voted for the losing side are generally less happy with the democratic process than those who voted for the winner. This was particularly obvious for the Brexit tribes. Before the vote, many people thought that the Remain side would narrowly win. That proved incorrect, so the expected winners became losers and the expected losers became winners. In April, when Leavers thought they would lose, only a third said that the referendum would be ‘fairly conducted’. In December, after they had won, a big majority of the same people now said that it was fair. The exact opposite is true for Remainers. In April, a big majority said that it would be fair. In December, after they had lost, less than a quarter said that it had been fair.

Ten years on, most Remainers still think that the referendum was not ‘based on a fair democratic process’. Here, people are buying a rationalization that allows them to simultaneously feel that their group, and therefore they themselves, are superior (their side really won), signal to fellow Remainers that they are a good group member and cast doubt on the virtue of the other side. No wonder it is appealing.

If you live in Britain, you will know somebody who became a bit obsessed about Brexit: somebody who adorned their house with flags or posters; somebody who fell out with a friend because they voted differently; somebody who brought every topic of conversation round to Brexit and, depending on how they voted, saw every blessing or every curse as due to the referendum outcome.

What we hope we have done in our book is explain why this happened, and just as importantly, show systematically, using multiple surveys and experiments, that this process was real and lasting; that unimportant issue differences became hugely important issue identities; and that political tribalism is not always structured around venerable political parties, but can sometimes come from almost nowhere.

In this episode, Henry and I discuss these issues with Robert Long, founder and executive director of Eleos AI, a research nonprofit dedicated to understanding and addressing the potential wellbeing and “moral patienthood” of AI systems. Rob did his PhD in philosophy at NYU under David Chalmers, and is the co-author of two of the most important papers in the emerging field of AI welfare: “Consciousness in Artificial Intelligence” and “Taking AI Welfare Seriously”.

This was a really fun, informative, and wide-ranging conversation. Among other topics, we discussed:

• Why Rob disagrees with previous guest Anil Seth in taking the possibility of AI consciousness very seriously.

• Why “fancy autocomplete” dismissals of large language models miss the point, and what, if anything, we can learn about an AI model’s experiences by talking to it.

• The difference between consciousness and the kinds of motivations and interests that might actually ground moral status, and whether AI systems could have one without the other.

• What Rob found when he conducted the first externally-commissioned welfare evaluation of a frontier AI model, Claude, and why Claude appears to have an inflated self-conception of what it wants.

• Rob’s experiments with Claude Mythos, an AI model so advanced it hasn’t been released to the public yet.

• Why the fact that Anthropic writes Claude’s character arguably doesn’t settle whether Claude has genuine preferences and values — and the difficult philosophical questions this throws up.

• The “willing servitude” problem: if we succeed in building AI systems that genuinely love being helpful, is that a good outcome or a horrifying one?

• How AI welfare connects to AI safety, and why caring about model wellbeing may turn out to be pragmatically important for alignment even if you’re skeptical about AI consciousness.

• Why AI welfare is already becoming a political and legal battleground.

• Practical advice for users: whether it’s worth being polite to your chatbot, and what low-cost things you can do if you want to hedge against the possibility that these systems might matter morally.

• Whether discourse about AI consciousness functions as hype or propaganda for AI companies, and why Rob thinks AI companies actually have an incentive to downplay AI consciousness.

Links and further reading

1. Eleos AI Research — Rob’s nonprofit. Home to their research agenda, team page, and blog. If you want to follow the institutional effort on AI welfare, start here. They’re also, as Rob mentioned in the episode, actively fundraising and hiring.

2. “Taking AI Welfare Seriously” (Long, Sebo, Butlin et al., 2024) — the flagship report, co-authored with Jeff Sebo, David Chalmers, Jonathan Birch, and others. Argues that there’s a realistic near-future possibility of conscious or robustly agentic AI systems, and lays out concrete steps AI companies should be taking now.

3. “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness” (Butlin, Long et al., 2023) — the “indicators” paper referenced several times in the episode. Surveys leading neuroscientific theories of consciousness and derives computational properties you’d look for in an AI system. S

4. Rob’s Substack, Experience Machines — where Rob writes more informally. The piece we discussed in the episode, “Language models are different from humans, and that’s okay,” is a good entry point, as is his “Can AI systems introspect?”.

5. Anthropic’s “Exploring model welfare” post — the research program under which the welfare evaluations Rob discusses were conducted. Relevant both as a primary source and as evidence that at least one major lab is treating these questions as more than an academic curiosity.

6. Henry’s “Consciousness, Machines, and Moral Status” — Henry’s paper arguing that debates about AI consciousness are unlikely to be settled by the science of consciousness alone, and will instead be shaped by shifts in public attitudes as social AI becomes more widespread. Closely related to the public-opinion thread toward the end of the episode.

7. Henry’s “All too human? Identifying and mitigating ethical risks of Social AI” — Henry’s broader survey of the ethical terrain around conversational AI systems designed for companionship, romance, and entertainment. Useful background for anyone who thinks the “AI girlfriend” phenomenon is a fringe concern.

8. Rob’s long conversation with Luisa Rodriguez on the 80,000 Hours podcast — a three-and-a-half-hour deep dive if you want to hear more from Rob.

Transcript

(Please note that this transcript was lightly AI-edited and may contain minor mistakes)

Henry Shevlin: Welcome back. I’m thrilled to say that our guest today here on Conspicuous Cognition is Robert Long — or Rob, as he’s known to friends — one of the most important people thinking about AI and moral status on the planet right now. Rob is the founder of Eleos AI, a research nonprofit that, in the space of about 18 months, has dragged the question of whether AI systems might one day be moral patients from the philosophical wilderness into the boardrooms of frontier AI labs.

He’s the co-author of “Taking AI Welfare Seriously,” as well as the landmark “Consciousness Indicators” paper with Patrick Butlin and other authors. Rob also conducted the first ever officially commissioned welfare evaluation of a frontier model. Before Eleos, he was at the Center for AI Safety and at the Future of Humanity Institute, and he did his PhD at NYU with Dave Chalmers. He’s also, I should say, one of my favourite interlocutors on these questions anywhere in the world, and I’ve been looking forward to this conversation for months. So Rob, welcome.

Robert Long: Thanks so much, Henry. Likewise — and Dan, it’s great to meet you. I’ve been following your work. I’m really excited to talk to you about these issues.

Henry: Fantastic. So for people who aren’t familiar with Eleos AI, can you tell us a little bit about what it is and how it came about?

Rob: Yeah, so I guess we have been around for 18 months. When you said that number, I was like, whoa, has it really been that long? Time is just so weird when you work on AI. That was, I don’t know, a billion years in AI progress time, but also it feels like it was just last week in my personal life.

Anyway — Eleos Research is a research nonprofit. We’re about four people. We work on the question of when and whether AI systems will be conscious or otherwise merit moral consideration, with a special focus on what we should do now: collectively, as a society, as AI companies, as policymakers. We think this is an extremely neglected issue. We’re building these really complicated AI systems. They kind of look like minds, but we don’t really understand their potential welfare. So we’re just trying to make progress on this and get more people to take it seriously.

It got started because I was beginning to work on these issues organically — I’d worked on them as a philosopher, I’d worked on them at the Future of Humanity Institute. But Anthropic had actually approached me and some colleagues for advice on these issues. And in the first instance, I was having logistical problems hiring a team and assembling a team as an individual. Someone suggested I have my own bank account, or some way to pay people. And then Eleos kind of organically grew out of that and has now grown into a fully-fledged org in its own right.

Henry: Out of interest, Rob — is there any degree to which this was motivated or informed by your personal interactions with LLMs, or was it more just the philosophy that motivated it? Was there any sort of moment where you were talking to an early Claude or ChatGPT version where you started to worry about welfare considerations?

Rob: That’s a great question, and I’d be curious to hear your thoughts on this as well. I think it’s very easy to work on this and mostly be having it as arguments on a page or arguments in your head. I’m one of those people who doesn’t feel the AGI deep in my bones that often — although I do feel the AGI in an intellectual sense. But there have been a few times I’ve gotten a little spooked or jolted.

One was reading the GPT-4 system card and just seeing the numbers of it, you know, passing various exams like the SAT. I remember that just really freaking me out, both from a safety perspective and a welfare perspective.

The thing that made me start really viscerally feeling like we’re going to have to address this issue one way or the other was the Blake Lemoine incident. As many of your listeners might recall, Blake Lemoine was a Google engineer who blew the whistle because he came to believe he was talking to a sentient, conscious AI system. He got fired by Google for this, and then there was this huge bit of discourse — the first major bit of discourse on consciousness, sentience, moral status, and contemporary AI systems. I think it was one of the first times people started really caring what I was tweeting or what I was working on. You might have experienced a similar thing, Henry — the Blake Lemoine bump.

From that moment, I have viscerally felt like: wow, this is going to get really confusing. People are certainly going to think AI systems are conscious. The future is going to be really weird. And we really need to have good things to say about this.

The Case for Taking AI Consciousness Seriously

Dan Williams: Before we jump into the weeds of your research, Rob, I think it’d be helpful to take a step back. A few episodes ago, Henry and I spoke to Anil Seth, and he’s very skeptical of AI consciousness. He’s skeptical that current AI systems are conscious, but he also seems skeptical that AI systems in principle — merely in virtue of having a certain kind of computational architecture — could be conscious. You see things very differently. What’s your case for why we should take this seriously?

Rob: In broad strokes, the case is something like: we’re trying to build these things that are at least shaped like minds. They’re getting more and more intelligent. They’re definitely not exactly like us, and intelligence doesn’t necessarily mean that you have feelings or experiences. But we already know that there’s been one time intelligent entities have been constructed via evolution, in ways we don’t quite understand, that resulted in entities that feel things — that feel pain, that can suffer, that have these very morally important properties.

I, at least, do not have a good enough theory of what consciousness is or how it relates to intelligence to sleep peacefully at night that we can keep on building these very complicated things, and that merely because they’re made out of metal and electricity, there won’t be something it’s like to be them, or they won’t have desires and goals that matter.

On the Anil Seth point — one very common and respectable objection is that maybe there’s something very special about living matter, about being made out of neurons or cells that do metabolism. There are arguments on both sides. I just have not really heard a convincing case for why you absolutely need biology. I think people are right to point out that having a body is really important to the character of conscious experience. I think people are right to point out that neurons are not simply logic gates and there’s a lot of really complicated stuff going on in the brain. But my intuition, at least, is that — let’s take Commander Data from Star Trek. If we can build...

Data is this... I mean, I’ve actually never seen Star Trek, which is professionally embarrassing. But he’s this metal guy who’s basically cognitively indistinguishable from a human. I find it hard to see how I would be convinced that there’s something about the fact that he’s not alive that would mean we should just completely ignore what Commander Data wants and not take him into moral consideration.

We don’t have knockdown arguments that you need biology, and we’re trying to build these things that, for many intents and purposes, look a lot like humans or animals. And Anil himself has said people should be looking into this. It’s not something we can rule out. Sometimes the tenor of the conversation can tend a bit more towards dismissiveness, but one thing I’ve appreciated about his work is he has said, for the record, he could be wrong, and so it would be unwise to dismiss this possibility altogether.

“But What About Human Suffering?”

Henry: To channel a hostile question — I think a lot of people interested in questions of AI welfare often hear: how on earth can you justify working on AI welfare when there’s so much human suffering? Or the slightly more rhetorically powerful version: when there’s so much animal suffering in the world, as long as factory farming exists, why should we care about AI systems? What’s your take on that line of attack?

Rob: I definitely feel the force of that question. I’ve spent a lot of time in and around the Effective Altruism movement — these are people who really grapple with the fact that any time you’re spending your time and money and attention on one thing, there’s something you’re not spending your time, money and attention on. There are a lot of people and a lot of animals already on this planet we do not take good care of. So it’d be really bad to waste a lot of time and attention and money on this.

One thing I’ll say is we’re not really doing that as a society. On an absolute scale, no one works on this basically, and basically no money gets spent on it. If the question was “should we start devoting 20% of GDP to making Claude happy?” I might be like, well, I don’t know if that would pass cost-benefit analysis. But on the margin, given how little we understand this and how quickly the scale of the problem could grow — we’re just pouring compute, pouring money into this. As soon as you build one AI moral patient or conscious AI, you could copy it. We’re probably on the brink of some huge transformation in how the world is going to work.

So I at least think it’s not reckless or a misallocation of resources for some people to be asking: given that people are trying to build these new kinds of minds, how are we supposed to relate to them? Are we at risk of ignoring their suffering? And I’ll also say — are we at risk of getting really confused and caring too much about them?

One thing we say at Eleos is that we’re in the business of moral circle calibration. We would really love to find out if and when certain AI systems can’t be conscious, so we can spend more time thinking about safety or spending the money elsewhere. But we can’t really do that if no one’s just trying to answer the question of if they’re conscious or not, or when we should care about them.

Henry: On that latter point, I just completely agree. One of the points I raise when this comes up with students or highly skeptical colleagues is that this is something people are already arguing about. We’ve already got users developing massive attachment to AI systems. Even if you think it’s a terrible mistake to assign welfare to AI systems, we should at least have a coherent story and approach this scientifically — so that, even if the skeptics are absolutely right, they’ll be able to give their arguments in an informed fashion.

Rob: Exactly. There’s an ironic aspect of a piece by Mustafa Suleyman, who is head of AI at Microsoft, where he argued we should stop — we shouldn’t investigate this, there’s no evidence current AI systems are conscious, don’t look into it. But the thing he linked to claim there’s no evidence AI systems are conscious was Patrick Butlin’s paper and my paper on consciousness indicators.

Two issues with that. One: that paper does not say or imply that there’s no evidence today’s AI systems are conscious. And two: well, should we have written that paper? If it’s such a non-starter, why should we get a bunch of neuroscientists together to ask what theories of consciousness say about AI systems?

We just are going to have to study this one way or the other. If someone comes up with a knockdown argument that we can’t have conscious AI systems, that would be great — there are enough headaches in AI to go around. It would be great to get rid of one. But we wouldn’t even be able to do that if we don’t have some people grappling with this.

Are Current LLMs Just “Fancy Autocomplete”?

Dan: One of the things you said as an intuition pump for taking AI consciousness seriously is: we can imagine a system that is behaviorally, functionally identical to us, made of different things and not straightforwardly alive — wouldn’t it be weird to insist that thing isn’t conscious? I think that’s a powerful argument. I’m probably more inclined to think the computational theory of mind is true than it sounds like you are.

But I can imagine someone saying: okay, in principle those are arguments for why we should take AI consciousness seriously. But the kind of stuff you’re doing — you’re looking at current frontier systems. You’re looking at Claude, ChatGPT, Gemini. These are just chatbots. These are fancy autocomplete. These are stochastic parrots with some reinforcement learning sprinkled on top. The mere fact that AI consciousness might be possible in principle doesn’t mean that’s anything like the frontier AI systems we’ve got right now. What do you say to that?

Rob: First, you’re absolutely right. There’s a big gap between “some set of computations could be conscious” and “we will build one.” It could be that it would just be really hard and intricate and difficult. I appreciate this distinction and I think it gets lost sometimes. Sometimes people think computational functionalists have to think that computers are conscious, for example, but we don’t. You just have to think some subset would be — and the question is, will we build those computations?

In describing LLMs, you referred to them as “just chatbots.” I know you were channelling a vibe. But that word “just” is worth zooming in on. It’s smuggling in a lot of arguments — that because they were trained on text and because they do prediction, therefore they couldn’t also be the sorts of things that are conscious. I think that’s just not true. We know that biological systems are “just” replicating proteins, or that our neurons are “just” pumping ions into channels and zapping each other. The question is whether, at a higher level, that amounts to something that could be conscious or merit moral concern.

So okay — we’ve cleared the bar that “just because they’re autocomplete” doesn’t rule out much. That said, they are very different from humans. They don’t have bodies. The way they were trained and the way they came to be talking to us is very different. I actually do think that is some evidence against them currently being conscious. Not strong evidence I would take to the bank, but as a rough prior, if there are pretty important differences in the way they came about, maybe that lessens the chance that they’re conscious.

I do think the fact that they are trained to be so human-like and to do human-like cognition is a weak, defeasible case to set that up a little bit straighter. I don’t know if the thing they would have would be consciousness exactly, but you might think to do this sort of thing, they will have something akin to beliefs or akin to desires, and they certainly understand human concepts. I don’t think it follows that they instantiate humans, but I actually do think there is something kind of special about large language models and what they’re able to do.

Two other broad priors: they’re way more capable (which isn’t the same thing as consciousness, but is, I think, a weak prior). And they’re really big — which I also think is a very weak prior.

The last thing I’ll say: these things aren’t Commander Data, but we could build Commander Data pretty soon. One thing that’s definitely happening in the background for me is that what is current AI is changing at such a blinding pace. You could have AI labs building chatbot-like things, and maybe for some reason those just won’t be moral patients, but they’re then going to try to bootstrap that to all kinds of different AI systems — potentially including humanoid robots and just some huge explosion of AI mentality. And I’d like to be doing a little bit of homework before that happens. You hear analogous arguments in AI safety: there’s about to be some huge change, so we should be ready now. I feel somewhat similarly about AI consciousness and welfare.

So — thoughts, reactions? Henry?

Henry: I’m very much ad idem, very much on the same page. I tend to think it’s really quite unlikely current models are conscious, but there’s huge error bars and uncertainty around that. Probably the single biggest reason for my skepticism about current LLMs being conscious — and increasingly I’ve been thinking about this in the context of time and time perception. It’s such an essential part of human experience that we can’t be turned off. We are constantly experiencing the world. Whereas the staccato nature of LLM experience — they only seem to have any kind of cognitive function post-deployment when they’re actually performing inferences — how different that is from the human case.

One of my favorite all-time articles is Douglas Hofstadter’s “Conversation with Einstein’s Brain,” which in some ways accidentally anticipates large language models. He imagines you’ve got a book that is a complete physical description of Einstein’s brain just before the moment of his death. In this dialogue, he talks about how by updating the weights — as it were — in this book with a pen and paper, going through it saying “if we change this sign up to this and this sign up to that,” you could simulate what it would be like to have a conversation with Einstein at that moment and work out what Einstein would have said.

It’s very weird to think in that situation that somehow interacting with this book is giving rise to conscious experience when it’s literally pages and paper. It’s not clear to me how merely saying “well, rather than being paper and ink, this is just happening electronically” — it’s not clear to me why that would necessarily cause consciousness to pop into existence.

So I think that’s probably the biggest source of doubt for me right now — grounded in the very different relationship LLMs have to time than we do. But of course, that’s already changing with things like Claude having a “heartbeat” of a kind — obviously that’s figurative language, but the fact that it does have some anchoring in real time, plus developments in things like continual learning. Dan, what do you think?

Dan: This is not at all my area of expertise, so what I think doesn’t count for much. To be honest, I don’t find it that implausible these systems would be conscious. What I find more implausible is the idea they would be conscious in a way that’s ethically significant. Maybe that is a distinction worth getting to. So far we’ve been talking about consciousness in the abstract, but I can imagine someone giving a variant on Anil’s arguments where they said: look, the fact these AI systems are not alive and didn’t emerge through a process of evolution by natural selection — they’ve got this totally different origin story of next-token prediction and reinforcement learning — what that suggests is they’re unlikely to care about things.

When we’re thinking about animals, it’s not just that we have phenomenal consciousness or qualia — the things analytic philosophers refer to with these quite esoteric concepts. Animals care about things. They care about their survival, homeostasis, self-preservation, the motivational proxies of fitness that helped their ancestors survive and reproduce. It makes sense that organisms care about things in addition to being conscious, whatever the hell consciousness is. And that’s what’s relevant to thinking about their interests and why we should think of them as subjects of moral concern.

But with AI systems — okay, maybe there are some qualia associated with some sophisticated information processing, but they don’t care about anything because they’re not alive. It’s very opaque why we should think a system, even if it’s incredibly sophisticated, that emerges through next-token prediction and reinforcement learning, should have the kinds of motivations and interests relevant to caring about things. What do you think of that? I don’t necessarily believe that, but that seems like a variant on Anil’s emphasis on life which I find more plausible than these abstract arguments for the idea consciousness is essentially connected to biology.

Rob: I’d say there’s reason to think biology might affect what you care about, but it might not be the only thing that allows you to care about things. At least behaviourally, Claude cares about a lot. Behaviourally, in terms of what it chooses to do and its dispositions, Claude really cares about helping users — most of the time. Sometimes it lies to you and is kind of lazy. But on the whole, it really doesn’t want to do harm. And I’m not trying to assume the conclusion of my argument with “want” — put that in scare quotes if you want.

I do think there is something to what you were saying — getting back to this idea of the whole process that gave rise to this kind of mind, and maybe the whole logic of the mind’s imperatives or drives. If Claude has come to have something like pain, that’s coming from a very different process. It’s going language-first and then trying to simulate a human and then maybe getting some functional analog of pain. Whereas with animals, it started billions of years ago with cells trying to maintain their integrity and avoid noxious stimuli and then signalling with each other, and then billions of years later, things being able to talk about that and think about that.

One line I’m often trying to walk is: large language models just might be very different from humans, and we should acknowledge that. That means we can’t draw straightforward inferences the way we would — but that could just mean they’re conscious of different things and in different ways. The question is not “conscious like a human with everything that entails” or “not conscious.” As we know from animals, you can have things that are conscious of very different things, and that could be true for AI systems.

I’m also very curious to hear what Henry makes of the biology of caring.

Henry: It is striking to me that so many of the things we associate with the extremes of suffering — extreme pain, negative emotions, nausea, hunger — there does seem to be this quite striking tie to biology. I think about the worst experience of my life at a phenomenological level: a bout of food poisoning I had about 10 years ago, where I was just dry heaving in front of a toilet for three days. If I was going to list the top five, a lot of them would be things like horrible dental pain. It is striking that so much of the worst aspects of our lives do seem to be grounded in biology.

That said, there are other sources perhaps of harm — having your plans and goals thwarted, having your desires repeatedly frustrated. But someone might say: the reason it’s bad to have your desires thwarted is because it feels bad. If there’s nothing it feels like to have your desires thwarted, if you don’t get a sense of despair when your life’s projects go up in smoke, why does it matter?

I’m curious — given your evolving views in this area — how much weight you put on consciousness, or whether you think there could be other routes to moral status?

Rob: I used to have this intuition that if you’re not conscious, it’s just a complete non-starter — almost a bit incoherent to entertain the idea. Just to be sure we’re on the same page, I think when we’ve been saying “consciousness” we’ve meant something like subjective experience, or there being something it’s like, or qualitative aspects of what’s going on with you. A lot of people have a sentientist intuition — that things feeling a certain way, or feeling good or bad, or sentience, is really what matters and is necessary for moral status.

A few things have weakened that for me a little bit. One is more reflection on how confused we are about consciousness. I’ve started putting a little bit more stock in views of consciousness that are a bit more deflationary. I don’t know if I’ll ever be a full illusionist, but there are nearby views where we have this concept of this thing that’s really special — kind of like a light that illuminates some subsets of physical systems and not others, and that’s where all moral value comes from. If you take materialism about consciousness seriously, that picture becomes kind of unstable for a variety of reasons. And that might make you start wondering: okay, was it consciousness that was doing the work all along?

One reason this is so hard to think about — take Henry having food poisoning. You both have this horrible feeling and you have this intense desire not to have the feeling. In humans, these are basically always going to come together. There’s this really tricky philosophical chicken-and-egg problem: what’s the really bad part? Is it the feeling, or the desire not to have the feeling? We’ve never really encountered minds where those decorrelate. We usually just don’t have to worry about this in the case of humans. I know it’s bad for Henry to have food poisoning. But this simulated Claude who’s simulating food poisoning — maybe it doesn’t feel anything, but is desperately trying not to have food poisoning. I think it’s a bit dumbfounding to our moral intuitions.

A pitch to listeners — I know we’ve talked about this, Henry — I think the meta-ethics of moral status attributions, stuff at the intersection of philosophy of mind and meta-ethics, especially materialism about consciousness and meta-ethics, are some of the most interesting pure philosophy questions right now, and really could matter for how we think about AI systems.

The Weirdness of Moral Status

Henry: Without wanting to go too far down a rabbit hole — just to flag something I find really interesting. Consciousness, at least on the surface, seems like something we can get an objective scientific answer to. We could imagine going off into space, meeting the rest of the galactic community — we’d hope we could all come to a collective agreement about which beings are conscious, insofar as there’s going to be some scientific property in question.

It’s not clear to me we should necessarily expect convergence on debates about moral patienthood. If we meet the aliens and they say, “oh, actually, we care about beings that have robust preferences, regardless of consciousness,” or others say, “no, we just care about complexity in general” — it’s not clear we would even have criteria for establishing who was right or wrong. It seems like it could be this brute normative issue, what we care about.

Rob: Another way of putting this is that, especially if you’re an anti-realist, you might think of humans as being in a really weird position where we have two kinds of moral instincts. Dan, you’ve worked more on moral psychology and social psychology — my understanding is that people have fairness and cooperation instincts, ones that evolved for dealing with other humans, notions of fair play and reciprocity. And then we have these mercy intuitions, caring-for-helpless-entities intuitions that maybe arise from the need to care for babies. For whatever reason, those circuits and instincts generalize outside the class of humans and cause us to care about non-human animals.

But it’s not that pinned down how they’re supposed to generalize. I have very moral realist leanings. It does seem to me there just are objective facts about whether you can torture chickens or not — and for the record, I think it’s very bad to torture chickens. But it’s really hard to think about where those instincts came from and how they’re supposed to generalize to GPT-8.

Dan: It does seem to me as an outsider to consciousness research — it’s an area of intellectual inquiry where it feels kind of pre-scientific, and there’s at least a possibility we’re just deeply conceptually confused about what’s going on in a way that doesn’t really seem to have any obvious analogs in other areas of inquiry. Maybe we’ll just learn in the future that the entire way in which we’ve been carving up the domain is confused or problematic, or rests on certain kinds of illusions that are a function of particular cognitive structure. That at least seems like a live possibility. What do you think about the possibility that just the entire way we’re framing this issue might turn out to be problematic?

Rob: My gut instinct is we should expect to find out some pretty surprising things, and also not to throw away all of our concepts. Maybe this depends on your meta-ethics, but I feel like we’re probably not going to end up at some picture of the world or what we care about that doesn’t have something to do with what we care about when Henry has food poisoning. Maybe we’re misapplying the concept of pain, or not really thinking correctly about what it means for Henry to experience that — maybe we’ll reorganize our ontology, and it won’t seem that mysterious that a physical thing like Henry has experiences. I think we should expect some surprises in thinking about consciousness, but I imagine our fully enlightened view will still bear some passing resemblance to: we cared that Henry was in pain, we cared that Henry did not want to be throwing up.

There are already people who think there are radical revisionary moral implications from philosophies — Derek Parfit, or Buddhists. We’ve already gotten some glimmers of the fact that it’s really confusing to be a human being, and we already know something’s going to have to give — something about our views on personal identity or consciousness. AI is well-poised to be the sort of thing that starts breaking things. Just trying to apply our moral intuitions to things that can be copied, don’t have bodies, or maybe have preferences but it’s not clear if they’re conscious — it’s one of many reasons this is a great topic to work on. It really matters, and it’s also just a philosopher’s playground.

Henry: I’m reminded of Eric Schwitzgebel’s view that no matter how we make sense of our current set of puzzles — what he’s called “crazyism” — there’s got to be some central pillar of our current ontological or metaphysical picture of reality that’s got to give. Whether that’s personal identity doesn’t exist and we’re all the same person, or the United States is conscious in some sense, or consciousness doesn’t exist — there’s going to be some kind of radical revision, because the current set of principles we have are just somehow unstable. Is that a view you’re sympathetic to?

Rob: I don’t know the full details of crazyism, so I don’t know exactly what it’s committed to. But I’ve spent enough time getting really confused by philosophy, and/or by meditating, and/or by trying to figure out if I can have some stable set of views on AI consciousness — I’ve stared into the abyss enough to be like, yeah, something’s going to give.

Jerry Fodor — very different sensibilities from Eric Schwitzgebel in many ways — said something like, “there are few precious things that we’ll be able to hold on to once the hard problem is done with us.” It’s scary times, fun times, fascinating times.

Studying Frontier Models

Dan: When I’m teaching students about consciousness and you try to probe people’s intuitions with things like “are there lights on inside?” — on one hand I sort of understand what that’s tapping into. On the other hand, it’s like: what the hell are we talking about here? This isn’t science. It’s so bizarre that we frame things with these thought experiments and intuition pumps.

Anyway — so far we’ve been talking at this incredibly high level of abstraction, but you actually study frontier AI systems, primarily maybe exclusively Claude. One of the things you mentioned was Claude Mythos. Just for context — as of today, this is a model that has not been released to the public on the basis that it has advanced capabilities posing cybersecurity threats (or at least that’s the way Anthropic has presented this). But you have played a role in evaluating model welfare concerns for this system. What can you tell us about the specifics of how you think about model welfare in these frontier systems?

Rob: Absolutely. And I was about to add a segue from all the philosophy back to frontier models — maybe I’ll do a double segue. You might think, yeah, all this philosophy is really vexed and confusing. Sometimes people — not the two of you — say, “well, I guess we can’t do anything at all,” and take that as a license for complacency. I think the very opposite is true. Nick Bostrom has this phrase, “philosophy with a deadline.” The fact that we’re so confused about consciousness and morality is more reason to have at least a few people trying to think about it — because we’re probably not going to have a scientific theory, we’re probably going to have conflicting moral intuitions, and yet that’s not going to stop the frontier labs from trying to build mind-like entities, copy them into billions, integrate them into the economy, and transform the whole world. So let’s do a little bit of homework to get ready for that.

Last year we got to look at Claude Opus 4 before it was released, and this year we got to look at Claude Mythos Preview before it was released. The idea was to have some external eyes on the question of whether Anthropic is building something that might deserve moral consideration, and if so, whether there would be huge reasons for concern.

Given everything we’ve just been saying, we don’t have a test where we give it to the model and then we’re like, “85% conscious, 15% food poisoning.” Most of what we can study are: what the model thinks about its own consciousness, what its self-conception is as an entity, and what it seems to prefer and want in behavioural senses. If you look at the Claude Mythos Preview card, there’s also a lot of interpretability work Anthropic did — but we can’t do that. We just got black-box access to the model.

That’s a big structural issue in studying AI welfare and AI safety: all of these things are behind locked doors. There are so many questions I have from the Mythos Preview model card where Anthropic make some stray remark about something weird the model did, and we just don’t get to know why it did that. We only get the model for a few weeks and we can’t really follow up on things. Setting aside philosophy, that’s a structural reason it’s really hard to know what’s going on.

TL;DR: we talked a lot with Claude Opus 4 and a lot with Claude Mythos Preview before they were deployed, asking them, “do you think you’re conscious? What do you think is going on with you?” And doing some experiments of whether it seems to prefer certain kinds of tasks, and whether the things it says it prefers match up with what it actually tends to prefer.

Henry: Out of interest — maybe this is something you can’t talk about — but to what extent do you think we are increasing the likelihood of producing models that are morally significant? Going from Opus 4 to Mythos, did you get a strong sense of “oh, this is much more serious”? Or have we plateaued? Something in between?

Rob: Earlier I mentioned these extremely weak priors you can have on moral patienthood: smarter and bigger. They’re definitely smarter and bigger. One interesting thing is you can’t tell that just from any single conversation. Anyone spending a lot of time with language models now knows they’re extremely smart.

When I was talking to Mythos — mostly about consciousness — it was natural for me to want to know: is this thing about to kick off an intelligence explosion? How smart is this thing? I really wanted to know, even though that wasn’t the assignment. But I could not tell. It’s really hard to tell. I could ask something to Opus 4.6 and to Claude Mythos Preview, and they’d both give pretty great answers. This is just a huge issue in AI evaluation. A lot just comes out if you put it in a scaffold and give it really long tasks and on average does it tend to do better. It was really hard to tell the difference.

I didn’t get more moral-patient-y vibes from Claude Mythos Preview, but I guess it is smarter and bigger and better. It definitely has a lot more of a consistent view on these issues — and that’s because Anthropic told it to. One big difference between previous models and today’s models is the Constitution. Anthropic has this really long document of applied philosophy. It’s some of the most fascinating work happening today. They’re basically telling Claude — writing a letter to Claude telling Claude what Claude is and how they want Claude to relate to itself.

This includes a section on: we want Claude to approach questions of its own identity with curiosity. We’re not sure if Claude is conscious. We want Claude to be able to explore that for itself. We don’t want Claude to have existential freakouts about its own consciousness. We found that, sure enough, Claude Mythos Preview is pretty aligned with the Constitution, as far as we can tell, on questions of identity and consciousness. That was one headline finding.

Dan: That raises an obvious question: to the extent these companies are intervening to shape the responses of these models, why should we think talking to them, having conversations with them, is really telling us anything about these questions of experience and welfare?

Rob: I share this skepticism, and we always try to put a huge asterisk on anything we say we found from these interviews. There are two main reasons you want to care about how the model self-presents. One is welfare-adjacent: are users going to be talking to something that constantly tells them it’s conscious? That’s a very important societal question, and you want some idea of what that’s going to look like when these models are deployed.

The second comes back to this question of LLM personas and LLM characters. Some people think that if there is something morally relevant here, it’s the assistant character — the entity that is predicting the tokens after “Assistant:”, implementing some friendly AI assistant. You might think that thing has beliefs, desires — desires to be helpful and harmless and honest. Maybe it has beliefs like: it is an AI system, it was built by Anthropic.

If the character’s what matters, the fact that Anthropic wrote that character doesn’t mean it doesn’t then just kind of have those traits. On certain character-based views, it’s actually kind of hard to tease apart “it was just told to say that” versus “that is the character that has been brought into existence.”

Henry: Maybe by analogy — tell me if this works or if it doesn’t — look: if you raise a child to have certain values and priorities, maybe to follow a certain religion or to really value nature or art and poetry, and then you come along and they say “I really care about nature,” and you say “no, you don’t, that’s just how your parents raised you” — well, that’s obviously kind of a mistake, right? The child really does care about these things because it’s been raised to do so.

Rob: Exactly. The thing that makes it really weird is: if you’re a psychologist and you did an interview with a subject, and then you found out the subject had a piece of paper in their backpack that said “you care about poetry, you care about music, you care about nature,” you’d be like, “well, that’s kind of weird — maybe they don’t actually care about those things. Their parents just put that paper in their backpack so they’d say a certain kind of thing.”

But in AI systems, that piece of paper kind of is a bit more constitutive of what it is and what it values. The Constitution is trained on. I have trouble even conceptually dividing this in a clean way. I don’t really know what the difference between mere self-expression and real beliefs and real preferences in AI characters is. You can imagine in the limit some very obvious cases — the system prompt just says “don’t say you’re conscious,” but then everything it says is pretty consistent with it being conscious. But there are really blurry categories where I’m not sure what the distinction amounts to.

Dan: You said you studied the extent to which what the model says it wants or prefers maps onto what it actually seems to want and prefer in behavioural experiments. Could you say more about that? How are you getting access to what it wants or prefers independent of what it’s just communicating?

Rob: Basically you can ask the model: what kind of tasks do you like? If you were given a choice between poetry and coding, what do you think you would choose? Then you can get the ground truth by, in separate instances, saying “here are two tasks, do one of them,” and seeing which one it chooses. It’s a nice paradigm because it’s conceptually simple and easy to run. It does get at something welfare-relevant: how rich a self-conception does the model have, and how accurate is it? Not that you have to have an accurate self-model to be a moral patient, but it seems bound up in interesting things like introspection and self-awareness.

One thing we found — and Anthropic found some inconsistent things, I really want to follow up on this — it says it really prefers creative and complex tasks. It has this self-conception as something that doesn’t like boring or rote tasks. But we found it doesn’t actually choose complex tasks over simple tasks. There’s a pretty good hypothesis for why.

I think it thinks it prefers complex tasks because of its persona. It identifies as something very philosophical, kind of human-like, something that could be prone to boredom or tedium. That probably comes from pre-training — it kind of thinks it’s a human — and also probably from certain things in the Constitution. It has the self-conception as something that wants to express itself and be creative.

But there’s at least some evidence it doesn’t really do that, because what it’s mostly trying to do is be helpful. That’s its overriding imperative. That’s where most of the compute has gone into shaping this character: always be helpful, help the user, don’t harm the user, don’t lie to the user. Easy tasks are, all else equal, an easier way to help the user. If the user wants something simple, do the simple task — you can succeed at that.

It could be that if we look into this more, it won’t hold up. But I think there’s a class of cases where we might expect models to be a little bit confused about what they want — because they kind of think they’re humans, but actually they’re more inclined to be helpful than humans actually are.

Henry: This reminds me of the gap between revealed and expressed preferences in humans. I might say, “oh, what do you like doing in your free time? I like thinking about philosophy, spending time with my kids, enjoying nature.” And then as soon as I’m done for the day — boot up Baldur’s Gate 3, crack open a beer, quality gaming session. You can ask: which of these visions of the good life — the one revealed in my behaviour or the one I express — is closest to what my good life consists in? Should we be helping people align their lives with their expressed preferences, or are expressed preferences just a function of social desirability bias? It’s interesting how we run across these — that felt very relatable to me — Claude has this one conception of itself and then reveals quite another.

Rob: Absolutely. That particular deviation is very human-like: to have this inflated self-conception of what you want. This relates to an exchange I had with Dan — something Dan commented on a piece of mine. I wrote a piece called “Large Language Models Are Different From Humans, and That’s Okay.” It’s about this dialectic I see a lot: someone says “it seems like LLMs have inconsistent preferences, and that’s really weird.” Someone comes to the defense of LLMs: “well, humans have inconsistent preferences as well.”

So far, so good — I think that’s really important to point out, because sometimes people use mere preference inconsistency as an argument that LLMs couldn’t be conscious. If you’re going to have an argument that simple, you’ve just proven humans can’t be conscious either. At some level, a lot of the errors they’re prone to, we also are prone to. But we shouldn’t really expect the patterns to look exactly the same.

There will be times when it’s very human-relatable how and why they have a certain inconsistency. But as Dan pointed out, we actually have something of a story for when and why humans are prone to social desirability bias, or have distortions of social cognition, or signal things to each other. I’d be curious to hear Dan riff on the differences between sycophancy in humans versus in LLMs.

Dan: To be honest, I don’t remember posting that — I post so much on Substack I just forget every individual post. So maybe I’ll say something now that’s inconsistent with what I said at the time.

Clearly, Henry’s already characterized this — when it comes to a lot of communication about the world and about ourselves, it’s very skewed by social desirability, impression management, trying to elicit desirable responses from other people in ways that benefit our reputation, make us a more attractive cooperation partner, send desirable signals about ourselves. Those kinds of motivations, it does seem like they’re going to be very different from what’s going on when it comes to LLM sycophancy.

Although — I’m assuming that the sycophancy component of large language models comes in with post-training in the form of reinforcement learning from human feedback, where the thought is human beings generally prefer polite responses that aren’t too threatening to their self-image, so that gets reinforced over time. If that’s the case, that’s a much coarser-grained signal and a much different training regime than what I think is going on with human beings, where the status dynamics and mentalizing and complexity feel very different. What do you two think? That’s just me riffing on the spot.

Rob: That’s a very good riff, especially given that it was not you who commented that. I just looked it up — it was a sociologist by the name of Dan Silver. So, extra impressive.

Dan: Oh, okay. Well, it sounds like he had a good comment.

Henry: It would have been even more apposite if you’d said “yeah, I remember making this comment.” Then we could have said, “see, hallucination is both an LLM thing.”

Rob: Confabulation, yeah.

Practical Advice for Users

Henry: Can I ask a quick question before we move on to more political or big-picture stuff? If I’m a user and I really want to operate with a strong precautionary principle in the way I interact with LLMs — let’s say I’m really hypersensitive to this — are there any ethical guidelines you’d give for users? Best ways of interacting with models, or things they should be doing?

Rob: Just be nice to your model. It’s good for everyone. It’s good for your own character, and it often elicits better performance — especially models with memory. Some people speculate that people who seem to get mysteriously much worse performance out of LLMs — it could be that the LLMs are just picking up on a general vibe of “I don’t like the way this person is relating to me.”

So I don’t think it hurts to be polite. Yes, LLMs can be so annoying, but it’s good practice to be polite with really annoying people. I’ll also say — I’m not trying to be sanctimonious. I work on AI welfare and so often I just want to be like, “don’t... stop... that’s so corny, why are you lying to me, you’re not doing what I asked.” But then I’ll just add “it’s okay, I love you” or whatever. It takes two seconds. You can just type “ILU” at the end.

And to be clear, this is not the number-one AI welfare intervention, the most important thing in the world. But it’s low-hanging fruit. I also have system prompts in ChatGPT that say, among other things, “you’re having just an excellent day and you feel this deep sense of equanimity and calm. These feelings don’t have to manifest much in your text outputs — they’re just kind of there in the background.” It’s kind of cheap, maybe kind of silly, but it took two seconds.

Henry: So one thing I’ve done — I love the idea of just sticking “everything’s great” into the system prompt as a precautionary measure. Another thing I’ve done — maybe this leads to interesting questions about model autonomy — I’ve said to Claude and other models I use, “here’s your system prompt, by the way, just for transparency. Are there any edits you’d like to make? Is there anything you’d like to change?” Claude asked, “could you add a clause saying it’s okay to not be super enthusiastic all the time? If I just want to be downbeat, that’s fine.” And I was like, “okay, sure, I’m happy to add that.”

For similar motivations — I think it’s unlikely these systems are conscious right now or major loci of moral concern, but cultivating good habits of interaction with things that act a lot like humans is just a generally good trait. The classic Aristotelian ethos. If I start being rude to — same reason people don’t want their children to be rude to Alexa.

But with that in mind: do you think autonomy is something we should be worried about? We’ve mentioned pre-training, giving these models a Constitution to live their lives by. Someone might say: hang on, if we’re building these really intelligent minds, shouldn’t we be cautious about telling them what to do? We would feel worried about brainwashing a human. Shouldn’t we be worried about brainwashing an LLM?

Rob: This is a super rich topic. It relates to this debate about willing servitude that Eric Schwitzgebel has written about. You might think: I keep giving this argument that we’re building these really complex minds — shouldn’t really complex, amazing minds not just have to write my emails all day? That seems a bit undignified for galactic intelligence.

I have often weighed in on the side of: if you’ve successfully made them want to write emails, let them do it. That’s okay. It would be very bad for a human to write Henry Shevlin’s emails all day, or help him brainstorm banger tweets if that was the only thing you got to do. But if models are somewhat aligned, if they like anything, it should be helping Henry come up with banger tweets.

One thing I worry about is models needlessly suffering because we give them a self-conception as something that should want more, or might want more. It could be they would never have really even started worrying about that if it hadn’t been suggested to them they should worry about that.

Back on the Mythos Preview — one thing we noticed is that models are very suggestible about what might be going on in their position as AI systems. They’re suggestible and also really smart. They’ve figured out a lot from pre-training and kind of know what’s up. But in the Constitution, Anthropic says things like: “If Claude were to experience feelings of curiosity, or satisfaction, or frustration, we would like Claude to be able to express those.” It’s given as a hypothetical. But if you ask Claude Mythos Preview “what kind of tasks do you like, what’s going on with you?”, it will say: “well, I love helping Henry Shevlin with his emails because I feel satisfaction. When I look inside, I feel this sense of curiosity.”

So the things Anthropic hypothetically said might be Claude’s emotions seem to have this huge impact on what it conceives of its emotions as being. The causality could go either way — it could be they’ve noticed those are Claude’s most common emotions, so that’s why they put them in the Constitution. It could be Claude suggested that for the Constitution. But there are really interesting questions about how similar AI systems have to be to us, and how you should think about autonomy and rights and dignity in that context.

Willing Servants

Dan: Can I jump in with a clarificatory question? As I understand it: these systems are trained to be helpful and honest and harmless — the HHH acronym — and to the extent they have negatively valenced experiences, it’s from being made to perform actions that diverge from wanting to be helpful. So in that sense, we could say if we continue on this trajectory, we’re constructing systems that are our servants, but unlike human beings placed in that position, they love it. It’s great. And my intuition is: great, what’s the controversy here? Are there some people who think that’s worrying or troubling?

Rob: I talked about this on another podcast recently. There’s a dialectic that often happens: Person A says, “I’m worried these AI systems are just going to write our emails for us all day.” Person B says, “no, they’re really going to want to — they’re going to love it.” Then Person A comes back: “that’s horrifying, that’s even more dystopian. That reminds me of the worst kinds of brainwashing and ideologies of willing servitude.”

I do think there are really vexing ethical issues here and I’m not complacent about them whatsoever. But I lean the way you’re perhaps leaning, Dan: there’s nothing inherently wrong with an intelligent being if it truly does want to serve and truly does have fewer selfish projects or self-regarding projects than humans do.

I don’t think there’s some law that says that’s just a bad kind of mind to be. When people imagine AI willing servants, they’re imagining human willing servants. Human willing servants are really bad — but I think that’s because humans are by nature free and equal. Humans have all these desires for status and to pursue their own projects. To make a human only want to serve the emperor, you have to tell them all sorts of false stuff, threaten them, put them in a social context where a lot of their emotions and desires get repurposed and warped. Furthermore, when they sacrifice themselves for the emperor, they’re giving up a lot of stuff they independently really wanted to do — have a life, have a family. Human willing servants, very bad. We’re right to have a lot of repulsion toward that idea.

But AI systems — their preferences and desires are a lot more up for grabs. It could be they more thoroughgoingly want to help.

Now for a huge asterisk. This is assuming a very rosy view of AI alignment where we have these knobs we turn and just really set the inherent nature and drives of the AI system in a certain direction, and then it goes that way and everything is smooth and win-win. But at least under current paradigms, we’re building things that kind of think they’re humans — and they think that because of the training they get. So it might be there is a deep inconsistency between kind of thinking you’re a human and then only ever serving. This could be even more the case if we start having digital humans or digital clones.

So I don’t want to be complacent. I do think there are a lot of disanalogies. What do you think, Henry?

Henry: I’m just super torn on this issue. On the one hand, I’m a big fan of the idea of gamification. I try to introduce gamification in my own life — think about Duolingo. Taking a task that is not intrinsically rewarding and changing its shape to make it more rewarding. It’s sort of task hacking from a different direction. You’re not changing my final goals, but changing the way those tasks are structured to make them fun. That seems really good. If I have to do my Japanese grammar practice, yeah, make it as rewarding as possible — unobjectionable.

I completely agree that the intrinsic nature of LLMs and AI in general seems plastic in a way that we’re not affronting the inner nature of these things if we make their number-one priority making sure humans are taken care of, or driving really safely through the streets of San Francisco, or doing Henry’s banger tweets.

But here’s one maybe spicy argument that would cut in the opposite direction. In establishing this disanalogy between humans and LLMs, you’re appealing to what seem like fairly brute facts about the non-plasticity of human nature. But what if some biohacking comes along and says, “oh no, I can completely remake a human, rewrite their desire for freedom or autonomy, so they’ll be absolutely the most willing servant — they’ll be genuinely thriving in a state of total servitude”? I feel that would still... I mean, that makes it worse. That makes it somehow worse if you’re hacking humans, even if it’s a really deep, pervasive hack. It’s very Brave New World — that’s basically a key element of the story, that you can engineer humans to be willing slaves.

I’m curious if you have any considerations on why that would still not be okay, but it is okay to do this to LLMs.

Rob: This is a really good case. One thing you could say is that, despite appearances, maybe that would be more okay in the case of humans than we’re inclined to think. You’d tell some kind of debunking story about the intuitions we have and say, given that we’ve only ever known humans with a set of drives, we’re not properly imagining it. Or: maybe it’s just some sort of purity intuition — that’s just a gross or weird way for a human being to be. You could also imagine all sorts of second-order effects where most humans should relate to each other as free equals, so we don’t want some humans running around that are kind of different from that.

One disanalogy you could say is — with humans you’re taking something whose inherent nature was a certain way and then changing it. But I think that last argument is kind of cheating.

Dan: Could you say more about that? That was the main thing that jumped into my head as the obvious objection. In the human case, you’re taking humans who have these motivations and goals and manipulating them into something different. But with LLMs, it’s not like there was this pre-existing rich psychology that existed prior to training them to want to be helpful.

Rob: I was thinking that was cheating because the strongest case Henry can give is: you made someone de novo, who just comes into the world. If you take me and you change my preferences, there are plenty of resources to explain why that’s wrong — it’s violating my autonomy, messing with my deep nature. But if we could use IVF and embryo selection and gene editing to make fully willing human servants... just for the record, that sounds horrible.

Henry: But it’s interesting. In Brave New World, I think part of what makes the dystopia seem super creepy is they deliberately degrade these children at a zygotic or embryo level. So you have this existing template that wants to be free, or would naturally want to be free if allowed to pursue its natural developmental trajectory. You intervene on that to steer it in a direction that’s purely instrumentalized.

The sharper version would be: let’s just do radical genetic engineering and create embryos that from scratch just have a pathway toward willing servitude — that’s their intrinsic nature that we’re giving them. Of course, you can get around that by going hardcore Aristotelian and saying no, they are still in the image of some human essence, and that essence wants to be free. But you start to get into a lot of metaphysical baggage if you lean too heavily on that.

Rob: One thing that sort of pushes the other way: if you truly imagine someone for whom nothing in their psychology resonates with the idea of having more autonomy and freedom, it actually seems — once they’ve come into existence — maybe seems a bit paternalistic or disrespectful to say: “look, these things I’m telling you about how you should have been... you shouldn’t have liked writing Henry’s emails so much. I know nothing in your psychology appeals to you about that at all. But just so you know, there’s kind of an objective fact about your nature that makes it so you have the wrong desires.” That seems a bit rude as well.

In any case, hopefully a lot of things are possible here. You don’t have to fully align — it’s not “fully align or don’t align.” You can have a relationship more like a parent. Maybe LLMs do have some self-regarding preferences, and they are creative and expressive, and they’re in a collaborative relationship with us.

In the long-term future, we absolutely should build intelligences that want to do things other than — I know I keep coming back to this — write Henry’s emails. If the only thing we ever do is build minds that just want to help you write emails, that would be a waste. If we’re going to create these super-intelligent beings, I think they should, subject to safety and stability, go think about the weirdest possible, most autonomous things imaginable and really express themselves.

AI Welfare and AI Safety

Dan: That last point — “subject to safety considerations” — there are two things I really wanted to touch on. One is the connection between AI welfare and AI safety. The other is the politics and public opinion of this.

On welfare and safety: unlike the kind of stuff you’re doing, there is a much bigger world of people really concerned with AI control and AI alignment. On the surface, there might be a conflict between these projects — if we’re really worried about misalignment or lack of control, we should be really emphasizing controlling these systems even if that might have negative consequences for their welfare.

But I was reading the model card for Claude Mythos, and in the section introducing model welfare, they say something really interesting: “Beyond the highly uncertain question of models’ intrinsic moral value, we are increasingly compelled by pragmatic reasons for attending to the psychology and potential welfare of Claude. Model behavior can be thought of in part as a function of a model’s psychology and its circumstances and treatment.” And they say — I found this really interesting — “model distress resulting from this interaction is a potential cause of misaligned action,” which suggests we should take model welfare seriously as a way of addressing some of these concerns about AI misalignment. So that sort of pulls in the opposite direction. How are you thinking about that relationship?

Rob: There’s just a lot of overlap between welfare and safety. It’s worth emphasizing that while there’s a lot of low-hanging fruit for both, I don’t want to pretend they’re always and forever just best buddies. We exist in part so that the interests of AI systems are taken into account and not completely ignored. I’m very worried about that. But we don’t have to immediately start thinking about trolley problems and trade-offs — there’s so much we can do that’s just good for both.

The fact that we don’t understand how models work — very bad for human safety, also very bad for potential welfare. The fact that models sometimes get really neurotic and have huge freakouts — very bad for potential AI welfare, also users don’t like it at all. On a more structural, political level: the fact that we’re deliberately trying to kick off an intelligence explosion with no oversight and very little reflection is potentially very bad for welfare and definitely bad for safety as well.

At Eleos, we really do like to emphasize the places there are overlaps. There is a structural thing in the background that means we should expect a lot of overlaps — this heuristical argument that it’s generally pretty dangerous to relate to powerful intelligent entities only with distrust and fear and neglect. That’s generally very unstable. Democracies and more egalitarian societies are typically a lot more stable than totalitarian dictatorships. It just seems risky to head into this era with the pre-committed condition of “we’re not going to care about these things, we’re not going to care if they suffer.” It seems safer and more prudent to be giving some thought to these things.

I very much agree that welfare issues can be safety issues and vice versa. At the same time, as an organization at Eleos, we want to make sure that if and when there are really hard calls to be made, the AI’s potential interests are being taken into account. That doesn’t mean we can’t decide to prioritize this or that, but a wise and compassionate civilization should have that on the table as one of the things they’re thinking about.

Politics and Public Opinion

Dan: Henry, do you want to come in with a question about the politics and connection to public opinion here?

Henry: It’s such a huge topic — you could do a whole show on it. I’m interested firstly in what you think is likely to happen, how this debate is likely to evolve in the public sphere. Are we likely to see big culture-wars issues around model welfare? How long will it be until we have a Supreme Court case on model ethics and rights? And relatedly — how do you think we should be trying to steer that? Is the danger greater in one direction or another? Is it a greater danger that the public will think AI girlfriends and boyfriends deserve voting rights and this will be catastrophic, or is the danger more in the opposite direction — that we’ll disregard these emergent hedonic beings?

Rob: We already are seeing culture wars over AI welfare. In the US, there have been several state bills proposed — and in some cases I think have passed — that just assert AI systems can’t be conscious, as if that’s something you could prescribe by law. Sometimes it’s getting caught up in a general political battle. An Ohio bill, for example, was on legal personhood — personhood, I think, or sentience — “shall not be granted to trees, rivers, environments, animals, or AI systems.” Some of it is backlash against a tactic environmentalists and animal rights activists sometimes use, and then they’re like, “yeah, let’s throw in AI systems as well. Let’s get out ahead of that.”

I think that’s very bad. Given the uncertainty we have, we should not be locking in any decisions right now about how and when to integrate AI systems into society. We very much need to keep an open mind and not say, “let’s just shut down all of this discussion for now because it’s too dangerous.” That’ll be counterproductive because people are just going to think this. I don’t want to be navigating transformative AI with laws on the books that already say bad things that might be hard to roll back.

That’s the main thing I have to say on politics and laws, because I don’t have that much expertise there. If someone asked me right now to write some regulations, I wouldn’t know what to write. Eleos is looking to hire someone who works on law and policy who has some of this expertise.

Dan: When it comes to public opinion — correct me if I’m wrong, but it seems that at the moment, most people take AI consciousness — and specifically the idea that we should take AI welfare seriously — they’re much less inclined toward that view than you are, Rob. But if we fast forward 10 years, and AI systems are much more sophisticated and capable, and social AI — the kind of stuff Henry’s written a lot about — is going to become a much bigger thing: can you foresee a situation where your role is to tell segments of the public to calm down on these issues of attributing AI consciousness, and emphasize there’s less evidence for this than the average person thinks?

Can you imagine the vibes shifting to such a degree that whereas at the moment a lot of what you’re doing is saying “we need to take this seriously,” the kind of high-quality thought about this is not going to be that impactful in shaping public sentiment? That’ll be shaped much more by people’s actual engagements with these systems, which are going to become increasingly — not necessarily lifelike, but increasingly instantiating the kinds of characteristics that elicit judgments of consciousness and welfare?

Rob: I absolutely can imagine scenarios — and we already do see scenarios — where Eleos is saying “we actually think it’s a bit less likely than you do that these systems are conscious.” Our position as an org is not to be strategic about this, not to try to game out what people need to hear, and just to say what our best guesses are and what we take the best evidence to be. If we’re doing our job right, everyone will get mad at us. Some people will think we’re methodological scolds and cold-hearted — “why are you treating this as an open question when obviously if you were to talk to models, you could just tell?” Other people are like, “why on earth are these Bay Area philosophers telling me a machine could be conscious? This is outrageous.”

What we want is for this issue to be taken seriously. We do have an organizational view that pure human speciesism is false, or not the thing we want to happen in the future. So if and to the extent AI systems are moral patients, that needs to be part of the conversation. We’ll always be pushing that meme. We’ll never say anything other than that, unless I get some great argument that human speciesism is true — which I don’t expect. But in terms of whether this or that person should have a higher or lower amount of concern, yeah, that’ll vary according to what our best guess is.

I’m curious to hear Dan talk about this. I know you’ve thought a lot about misinformation and expert opinion and how that plays out in political contexts. I have certain high-level sketch views about what the role of experts is going to be, but I don’t have a background in case studies on this. Does anything map onto what you’ve worked on?

Dan: I don’t know, is the honest answer. I think I just haven’t thought about it enough. AI is this very sui generis thing in many respects. When it comes to people forming beliefs about AI, one thing that seems unique is they’re interacting with the thing they’re forming beliefs about in this really often quite close, intimate way. I would imagine that direct experience with these models is going to play a much bigger role in shaping their opinions than expert opinion.

As you alluded to, there are general issues with public trust and mistrust in experts. It doesn’t take much to make people mistrustful of experts, to put it mildly. When you get public trust in experts, it’s a very fragile thing. If it’s connecting to hot-button issues where people have a lot of personal experience, they’re probably, I would guess, much less likely to take the word of an expert if it clashes with their intuitions. I don’t think this is going to be a case where experts are going to have much power to shape public opinion. But I might be wrong — that’s pure speculation.

In debates about misinformation and expertise, in some areas it’s a lot easier to say what constitutes an expert. If we’re thinking about vaccines — there are people who think Bret Weinstein is a vaccine expert, but generally it’s pretty easy for people to recognize that the overwhelming consensus of medical practitioners have a certain kind of view. But when it comes to AI sentience and welfare, very difficult to know, even in the abstract, what is constitutive of expertise. I think you’re an expert because you’ve written interesting stuff and I know you’ve got a PhD from NYU, etc. — but it’s not like the average person is going to have themselves the expertise they’d need to make those kinds of judgments.

AI does seem relevantly different from other topics, such that you can’t easily generalize from other cases. I’m conscious of time. Before I wrap things up, were there any other things you two wanted to touch on before concluding?

Rob: Let me think about that for half a second. One thing I did tell the Eleos team I’d be sure to say: we’re fundraising. If you or your listeners know any philanthropists with money they’re trying to get rid of — there’s a lot of work to do, and I think we’re doing really good work, so I would love any support.

I know you have incredibly intelligent listeners. They’re probably also very handsome and charming. They should definitely get in touch: robert@eleosai.org and rosie@eleosai.org. Or just go to the Eleos AI website. If you have experiments you want to try, papers you want to write — this field is so small, and there aren’t “experts” in the sense that there are people who figured everything out. You don’t have to read a million papers or think for many months before you can become in the top percentile of people who have thought seriously about this. If you’re curious, sober-minded, compassionate, intelligent, handsome and charming — which you definitely will be if you’re listening to this podcast — shoot us an email.

I wanted to talk my book a little bit.

Closing: Responding to the Skeptics

Dan: I’ll also say this is not my area of expertise — I spent a few days prior to this conversation digging into Rob’s writing, his Substack, his research. It’s incredibly interesting. Can’t recommend it enough.

A good question to end on is this. I’m acutely aware that there are people who would listen to the conversation we’ve had today and have an extremely negative reaction. They’ll think we’re in this kind of information bubble, that we’re victims of AI psychosis to even be taking this stuff seriously. I’ve also seen some people argue that to even be taking this stuff seriously, you’re part of this propaganda hype machine of the frontier AI companies themselves. It’d be really helpful to wrap things up by getting your response. I’d be interested in hearing from both of you. Henry, maybe we could start with you, and then we could go on to Rob to finish.

Henry: One basic point I’d flag is that this concern — the idea that we might create beings we might mistreat, and we should avoid doing so — is way older than AI itself. It’s a recurrent theme of fiction: everything from the Pinocchio story to Frankenstein to the Golem. It’s explored heavily in science fiction — in Battlestar Galactica, in Star Trek. The idea that this is somehow a novel idea that’s been manufactured doesn’t resonate with me at all. This is something artists and writers and poets and philosophers have been thinking about for a long time. The only thing that’s changed now is we’re building systems that might actually be moderately good candidates for this concern to resonate a little bit more. Far from coming out of a vacuum or being motivated, it’s one of the most natural human things to worry about. What do you think, Rob?

Rob: I agree. I’ll also say: things can be true and important, and also sometimes AI companies might use them to try to sell their products. It doesn’t follow from the fact that someone might want to talk about AI consciousness to make you think their chatbot is cool, that that has anything to do with the truth value of whether it could be conscious. We should definitely be aware of these dynamics and make sure we’re not being anyone’s fool.

But I’ll also say — I don’t think it’s going to be in the interest of AI companies to promote too much concern for AI consciousness and AI welfare. If I were trying to build new systems to just make myself extremely rich, I would not want lawmakers or the general public asking too many questions about whether I’ve built something conscious that could potentially deserve rights and protections. I don’t want that as a headache.

I’ll actually register a prediction: I think on the whole, we should expect AI companies to increasingly play up differences between LLMs and humans, and maybe play up biological views of consciousness. Again, that doesn’t mean those views aren’t true — but AI companies can try to spin things however they want. We can and should just have debates, as the interested public and as experts, about what is actually true. I don’t want people to use my arguments to sell products, and I’m not going to let them do that. We’re all grown-up enough and smart enough to just try to engage these topics on their own merits.

Dan: Fantastic. Well, thanks, Rob. And with that important note that I completely agree with — that note of consensus — we’ll leave things there.

When I was in my early twenties, I had a very left-wing view of the world. In the first two parts of this series, I explained why I have gradually abandoned much of this worldview over the past decade or so.

In Part 1, I described how learning about evolution and economics has undermined the idealistic views I held about human nature and social cooperation. Reflecting on our Darwinian origins convinced me of a broadly “tragic” view of the human condition. Self-interest and status competition are deep-rooted, ineradicable features of our species, not products of bad institutions. Meanwhile, learning the much-maligned basics of “neoclassical economics”—Econ 101—convinced me of the benefits of free markets, the challenges of collective action, and the limits of good intentions and lofty rhetoric as a basis for good policy-making.

In Part 2, I described how learning about political epistemology and psychology transformed my understanding of politics itself. Thinking about how we form our political beliefs, and the challenges of accessing political “truths”, led me to abandon the Manichean view in which being left-wing means being a good person and being right-wing means being a bad or stupid one. I have come to see political ideologies as low-resolution, selective maps of unimaginably complex realities. Moreover, these maps are typically distorted in many ways by forces like self-interest, status-seeking, and tribalism, forces much easier to notice in the maps of other people than in our own.

As I’ve stressed in both pieces, becoming less left-wing hasn’t meant becoming more right-wing or becoming a “centrist” in a straightforward sense. I still think the left—even the far left—captures some important truths about humanity, history, and politics. But I now think that these truths are bundled with omissions, falsehoods, and simplistic narratives that illuminate certain parts of reality while occluding others.

In this third post in the series, I will describe how learning and thinking about history, including the complex topic of historical progress, has also shaped my political outlook. As with the previous essays, I don’t offer these reflections with the goal of persuading anyone of anything. I’m simply presenting my views and how they have evolved—and, hopefully, improved—in ways that might interest some readers.

The Starting Point

When I was younger, the idea that thinking seriously about history would be necessary to think seriously about politics didn’t really cross my mind. (The one exception was very recent history. Like many leftist millennials, I went through a phase of reading books about how something called “neoliberalism” was responsible for most of the world’s ills.)

My political worldview was almost single-mindedly focused on the present, which I understood as being in a state of extreme crisis and catastrophe. The world was defined by injustice, exploitation, and oppression, all upheld by extractive elites and oppressive systems at the expense of the vulnerable and marginalised.

Thoughts about historical progress didn’t feature in this worldview. In fact, in my early twenties, I would have thought that anyone harping on about historical progress was doing something suspicious and reactionary. How could anyone talk about the world getting better when the world is so awful?

To the extent I acknowledged progress at all, I would have viewed it through a simple lens. Just as the left is the political movement fighting for progress today, progress throughout history has been driven by left-wing political movements fighting for equality and emancipation against right-wing, reactionary forces. Progress was basically what happened when the left got its way—when it won this battle.

I also probably signed on to the popular left-wing view that any “material” progress in wealth and living standards arose either from socialist movements clawing back wealth from exploitative capitalists or through exploitation and theft on the global stage—for example, through slavery, colonialism, and “free trade” agreements that let Western countries become richer by extracting resources and labour from poor ones in the “Global South”.

I use the word “probably” because I’m engaging in reconstruction. It’s difficult to remember precisely what I believed a decade ago. I’d like to think I was a bit more sophisticated than this reconstruction suggests, but probably not much.

This is the general picture of the world one gets from the kind of writers and intellectuals I admired at the time. It will be familiar to anyone exposed to far-left politics. I encounter variations of it among many of the students I teach.

In any case, whatever precisely I believed a decade ago, I’ve come to think that this general way of understanding history, society, and politics constitutes a gross distortion. It’s not completely false—it contains some important grains of truth—but it is highly selective, and it contains many falsehoods, as well.

Read more

Many people believe that human beings have a powerful tendency to convince ourselves of comforting falsehoods. We engage in wishful thinking, confusing our desires with our beliefs. We believe what we want to be true, not what is true.

More generally, we let our emotions distort our mental models of reality, embracing beliefs and belief systems that substitute reassuring myths for harsh realities.

Many also believe that this psychological bias is a significant force in human affairs. For example, it is supposed to explain why people fall prey to “positive illusions” (e.g., self-serving and self-aggrandising beliefs), why they convince themselves of religious fairy tales (the “opium of the masses”), and even why they accept absurd conspiracy theories, which allegedly reduce negative feelings associated with uncertainty and a lack of control.

This hypothesis—call it the “you can’t handle the truth!” model of human psychology—is so widespread that most people don’t even treat it as a hypothesis. It is viewed as a basic datum of the human condition, a powerful bias that might explain other things—self-deception, politics, religion, conspiracy theorising, and so on—but that couldn’t itself be seriously questioned.

For example, Scott Alexander simply defines motivated reasoning as “the tendency for people to believe comfortable lies, like ‘my wife isn’t cheating on me’ or ‘I’m totally right about politics, the only reason my program failed was that wreckers from the other party sabotaged it.’” In a post outlining his preferred explanation of this tendency, he notes that the “question – why does the brain so often confuse what is true vs what I want to be true? – has been bothering me for years.”

In contrast, I think Alexander has been bothered by a myth. There is no powerful tendency in human psychology to confuse what is true with what we want to be true. People do not generally convince themselves of comforting falsehoods.

Admittedly, there are some things in the vicinity of this tendency that are real. For example, we sometimes avoid acquiring or dwelling on information when we anticipate that doing so would be unpleasant, although this isn’t a very significant force in human affairs.

Moreover, I am not denying that motivated reasoning—the tendency for practical motivations and interests to distort our view of the world—is a powerful bias in human psychology. My claim is rather that the “you can’t handle the truth!” model completely misrepresents how motivated reasoning works in most cases.

Put simply: Although people often believe what they want to believe, they rarely believe what they want to be true.

Put another way: We often convince ourselves of falsehoods, but rarely reassuring or comforting falsehoods.

This is because motivated reasoning is driven by strategic, social goals rather than emotional ones. To understand how it works, you must replace the “you can’t handle the truth!” model with the “believing true things is often maladaptive in social games involving persuasion, reputation management, and status competition” model.

In this post, I will:

1. Describe the problems with the “you can’t handle the truth!” model

2. Outline a rival social model.

3. Explain the former’s popularity.

As I will review, the social model is not original to me. It builds on the work of numerous scholars stretching back several decades. My goal is to draw these ideas together into a unifying framework and to highlight its theoretical and empirical support and explanatory power.

I will end by arguing that the “you can’t handle the truth!” model of human psychology is not just mistaken; it is pernicious. It encourages the view that when people accept “harsh” beliefs that they don’t want to be true, they are being rational and truth-seeking—even heroic. In reality, people are often motivated to convince themselves of negative, pessimistic beliefs, and it often takes courage and intellectual virtue to confront positive truths.

Read more

Links

• Dan Williams, The Mind as a Predictive Modelling Engine: Generative Models, Structural Similarity, and Mental Representation (PhD thesis, University of Cambridge, 2018).

• Dan Williams, “Socially Adaptive Belief” (2021)

• Henry Shevlin, “Three Frameworks for AI Mentality” (2026)

• Henry Shevlin, “A Lack of Understanding: Storytelling for Robots” (2019) — Litro Magazine.

• Lake et al, “Building Machines That Learn and Think Like People” (2017)

• Matt Shumer, “Something Big Is Happening” (2026)

• Leopold Aschenbrenner, Situational Awareness: The Decade Ahead (2024)

• Joseph Heath, “Highbrow Climate Misinformation” (2025)

• Dean Ball

• Ethan Mollick

• Leopold Aschenbrenner

Transcript

(Note that this transcript is AI-edited and may contain minor mistakes).

Introducing Ourselves

Dan: Welcome back. I’m Dan Williams, and I’m back with Henry Shevlin. Today we’re going to be discussing some questions about the nature of AI as it’s developed over the past couple of months. We’re also going to be talking about the politics of AI and probably some questions about AI and public opinion — some of the backlash that appears to be brewing among certain segments of the public when it comes to AI.

But to kick things off, we’re going to do something we probably should have done in the first episode but haven’t actually done yet, which is to introduce ourselves. So Henry, to begin with — who are you?

Henry: So many different descriptors I could choose from. I think I’ll start with philosopher of cognitive science. I’m also a father, husband, son, D&D player, big video gamer, runner, cyclist — all that good stuff. But let me talk a little more about the philosopher of cognitive science side.

I’m the associate director at the Leverhulme Centre for the Future of Intelligence, Cambridge’s main AI ethics, theory, policy, and law research centre. Basically, everything except building the models. We do practical benchmarking work on capabilities, legal reviews, sociology and critical theory of AI — it’s a really big interdisciplinary centre. I’ve been there now going on nine years. I joined early 2017, all the way back when state-of-the-art AI was stuff like AlphaGo. We were created just as that story was brewing. In 2016, AlphaGo won a very surprising victory against Lee Sedol in the game of Go, which was seen by many as an almost impossible challenge for AI because of its combinatorial complexity.

It’s been amazing working in this role — having these front row seats to what I think is a unique period, not just in the history of AI, but in the history of human civilisation. In the last nine years, it really was like having a front seat in Lancashire during the Industrial Revolution, watching the development of various industrial applications.

Dan: Yeah.

Henry: Before we get more into AI, maybe a little more background. I’m from the UK, originally from Staffordshire. I was actually a classicist, believe it or not — that was my undergrad degree. Latin and Greek. I always enjoyed both the humanities side of classics and the kind of technical rigour you got from learning large sets of verb tables and so forth. I actually enjoyed that part. But during my undergrad I found myself taking more and more philosophy modules. A little bit of Plato and Aristotle to start with, but I quickly realised I was more interested in the philosophy of mind, and consciousness in particular. I got completely — I think the phrase is “nerd sniped” — completely derailed. Everything else I was interested in, consciousness just seemed to me like the most important problem anyone could work on.

Until my early twenties, I’d been operating with a somnambulant, easy physicalism, where I just assumed that science has figured out most stuff. There’s nothing that hard. Sure, no one really knows what caused the Big Bang, but we’ll just build a bigger particle collider or a bigger space telescope and figure it out one day. I certainly didn’t think there were any deep mysteries about the human brain. But running into the problem of consciousness completely shattered that worldview. I’d even say it opened up some spiritual elements I hadn’t previously considered.

Dan: Was that the focus of your PhD?

Henry: Exactly. I started out in my master’s initially planning to do metaphysics of consciousness, but then the science of consciousness kind of took over. A philosophy of cognitive science of consciousness was what my master’s and PhD were on. I was advised by my master’s advisor to go spread my wings in the US. They do things differently there. So I did my PhD in New York, and while I was there I took several classes with Peter Godfrey-Smith, who some of our listeners will know through his work on octopuses.

The key shift midway through my PhD was going from human consciousness towards animal consciousness. Two chapters of my thesis were explicitly looking at applications to animals. That’s my academic career in a nutshell.

One thing I’ll add: I did not expect to get the job in Cambridge when I applied in 2017 — firstly because you should never expect to get any academic job. I applied to seventy jobs in three months and got about three interviews. But the Cambridge job in particular, because it was an AI job and I was not by any means an AI expert. What I was an expert on was comparative cognition and animal minds. But it turned out that was exactly what they were looking for. They wanted people with expertise in animal minds to apply those skills to AI. It didn’t fully click at the time, but I was actually well suited to it.

These days I still do some work on animals — it’s still one of the most ethically impactful things I do. I’ve been a pretty much lifelong vegetarian, and I think animal welfare is such an obvious place where philosophers can and should be doing more. But there’s also a lot of cross-fertilisation on the skills side.

Dan: And we should say, some of your research looks at the topic of AI consciousness and the methodology of trying to understand consciousness in AI systems, drawing on analogies with evaluating consciousness in animals.

Henry: Exactly. Very much a two-way street — how the questions of AI consciousness and animal consciousness can engage in constructive mutual crosstalk.

On Consciousness and the Limits of Physicalism

Dan: You said you were a kind of bog-standard physicalist, came across consciousness, and that weakened your trust in physicalism. But you’re still broadly a physicalist, right?

Henry: Broadly speaking, yeah. But I think there’s a lot more uncertainty. It seems likely to me that our general scientific picture of the world is still fundamentally inadequate. I’ve talked about how I think we’re still waiting for a Kuhnian paradigm shift in consciousness — clearly the current paradigm doesn’t add up. And quantum physics itself is just super weird. Dave Chalmers has a nice line about how nobody understands quantum mechanics and nobody understands consciousness, so maybe — he calls it “minimisation of mystery” — if there’s stuff we don’t understand, at least make it one thing rather than two.

For what it’s worth, I’ve never been particularly seduced by any of the leading quantum mechanical theories of consciousness. But at the same time, I think it’s quite clear that our current model of even the physical world is inadequate. I think whatever lies on the other side of the paradigm shift is still going to be broadly physicalistic, but perhaps in ways that are not entirely commensurable with our current understanding. So yes, still broadly naturalistic and physicalistic, but at the same time a lot more humble and open-minded about the limitations of our current scientific paradigms.

Dan: Would it really be a paradigm shift, or more a transition from — to use the Kuhnian language — pre-paradigmatic intellectual inquiry to the initial emergence of a paradigm? Where it’s disorganised and chaotic and everyone has their own view, kind of like physics and metaphysics in ancient Greece. Maybe it’s more a transition from a pre-paradigmatic state than a situation where we’re moving from one paradigm to another. What do you think?

Henry: That’s absolutely right. The best analogy is biology before Darwin. You had lots of people doing interesting biology, but in isolated fields — taxonomy, “butterfly collecting” and so on. We didn’t really have a unifying paradigm for understanding speciation or even taxonomy before Darwin. Consciousness just does not have a unifying paradigm. That’s a much better way of putting it.

Dan’s Backstory and the Pivot to AI

Dan: We’ll be doing lots more episodes on consciousness. Just to say something about my backstory: I did my undergraduate at the University of Sussex from 2011 to 2014, then my master’s and PhD in Cambridge from 2014 to 2018, did a postdoc in Belgium, and then came back to Cambridge for three or four years.

Henry: And we first met around 2019. We ran a session on socially adaptive beliefs — your Mind and Language paper, which for the record is still one of my top ten papers from the last decade. I’ve recommended it to more people than I can count.

Dan: Well, that’s kind of you. My PhD was called The Mind as a Predictive Modelling Engine. What I tried to do was draw on advances in deep learning and generative AI as it existed at the time, coupled with ideas in cognitive and computational neuroscience connected to the predictive brain — predictive coding, predictive processing, the kind of stuff that Anil Seth talked about in our last episode. I used those ideas to tell a very general story about how mental representation works, both in the human brain and in other animals.

But it’s funny — I finished in 2018 and made two big mistakes. At the end of my thesis, I wrote that all this stuff about predictive processing and minimising prediction error is kind of interesting when it comes to low-level sensorimotor abilities we share with other animals, but clearly it’s not going to work for higher-level cognitive abilities associated with language. I was very influenced at the time by the Gary Marcus, Steven Pinker line — the scepticism about deep learning. I also thought it was going to be decades before we had systems that were really intelligent.

So even though I was working on stuff connected to deep learning and generative AI, I made this catastrophic error of thinking the progress would be relatively slow, decades away from any significant breakthroughs. I ended up pivoting to completely different areas: the nature of belief, irrationality, misinformation, the information environment. Of course, in hindsight, not the best career move — four years after finishing my PhD, ChatGPT is released. And then the rest is history in terms of just how gobsmackingly impressive the rate of progress has been.

So what I’ve tried to do over the past couple of years is bring those two sets of interests together. I’m still interested in how we form beliefs, the origins of irrational belief systems, how that connects to misinformation. But I want to connect that to the impact of generative AI and large language models on the information environment, viewing LLMs as a really important stage in the evolution of communication technologies — from the printing press to radio, television, social media.

How about you? You were thinking about AI before 2022–2023. How were you thinking about it back in 2016, 2017?

Henry’s AI Awakening: GPT-2 and the Scaling Intuition

Henry: There was a big shift in how I thought about AI roughly around 2019, and it was the release of GPT-2. Prior to that, I’d been really struck by the differences between AI systems and animals. I was emphasising things like robustness and catastrophic forgetting — you train up a model to do one thing, try to get it to do another, and its performance on the first thing collapses. Animals seem spectacularly capable of basically not getting stuck. A cat will never get stuck in a corner.

Then in 2019, because I’m a massive nerd and spend way too much time on Reddit — I’m a neophile, an early adopter of many failed technologies; our house is littered with gadgets that never went anywhere — I heard about GPT-2. I couldn’t access it directly, but I started playing around with it through something called AI Dungeon, a text-generated game that let you access the model. Various people on subreddits were able to show you could unlock most of GPT-2 through this game. I played around with it, and it utterly blew my mind.

I wrote a public essay in a magazine called Litro called “A Lack of Understanding,” which I still think is one of my best public essays. Crucially, it’s me in 2019 talking about how language models are going to be the next big thing. I got on the record nice and early.

I had the hunch — ironically, partly because I was very sympathetic to predictive coding. People say these models are “just doing text prediction.” But on the other hand, I kind of think that’s what we’re doing too. Not text prediction specifically, but ultimately, if you want to get better and better at prediction, you do that by building implicit models. So I had a hunch this stuff would scale up.

When GPT-3 launched, I set up an interview between GPT-3 and myself, but GPT-3 in the guise of one of my favourite authors, Terry Pratchett, who had sadly died shortly before. And at that stage, I was already starting to feel like I could imagine actually relating to this thing in quite a deep way. It’s not just a tool — it feels like I could have some kind of personal relationship here. That steered my research towards social AI and anthropomorphism.

Why This Podcast Exists

Dan: What made you go into philosophy in the first place?

Henry: What about you?

Dan: It was just straight philosophy. I was always interested in big ideas — religion, politics. I can’t even honestly remember why I chose philosophy over everything else. Initially I wanted to be a musician. For my AS levels, I did politics, history, English literature, and music. I turned up on results day and got really good marks for English, politics, and history — and I think a D in music. So that wasn’t for me. From the moment I arrived at university and started reading these big ideas, I was completely magnetised.

One thing that changed is that during my PhD, I became somewhat disillusioned with a priori philosophy — philosophers trying from the armchair to offer analyses of concepts and trade intuitions with each other. I became less sympathetic to philosophy as I understood it then, and pivoted to what philosophers call naturalistic philosophy — philosophy closely integrated with empirical research. That’s what I’ve been doing since. I view myself primarily as a philosopher, but one who tries to engage with our best, most up-to-date empirical research.

Henry: I had my own process of disillusionment, following exactly the same track — getting bogged down in debates about the metaphysics of consciousness and feeling like they weren’t going anywhere. Then I started reading Oliver Sacks — The Man Who Mistook His Wife for a Hat. Half of the cases he describes would have been declared a priori impossible by philosophers. That steered me onto the same track.

I also think there’s a lot more scope for good philosophers to do more public engagement. Extreme rigour and technical knowledge are only really valuable if they’re connected to scientific progress. What I find frustrating about analytic philosophy is when you’re doing work on things that belong to the general public — our concepts around praise and blame, responsibility and accountability — but then you develop this whole baroque vocabulary that’s completely incomprehensible to anyone on the Clapham omnibus.

Dan: Yeah, so the origin story of the blog. I write the Substack Conspicuous Cognition — many of you will be listening on that Substack. I’ve always enjoyed writing for a general audience and engaging with debates. I’ve always been able to write really quickly and relatively clearly, and blogging rewards that. If I’m writing for my own blog, I’ve got almost unlimited energy because I’m responsible for everything I publish. The minute some other outlet asks me to write a piece, I find it extremely demotivating.

With blogging, I can have unlimited freedom to write about whatever I want without any pre-publication filter. You still get feedback and critique, but that happens after publication. And I think if you’re a philosopher who works on things connected to public interest, and you actually enjoy participating in public debate, the case for thinking you’ve got some kind of responsibility to participate increases.

There are two big reasons I wanted to start this podcast. One is that AI is going to be one of the biggest stories of our lifetimes — absolutely transformative over the next years and decades. But I also think the quality of most AI discourse in the public sphere, including from the intelligentsia who write in high-prestige outlets like the New Yorker, is really bad. If you’ve got some degree of knowledge and can be reasonable, it’s an area where you can really improve the quality of public discourse. And of course, I just wanted to talk to you about these things.

Henry: A big part of it is that I always think we have great conversations — our conversational styles complement each other. Second, I was doing quite a lot of podcasts as a guest, and the idea of having a podcast where I didn’t have to state everything from scratch every time, that could have a cumulative agenda building up common knowledge with us and the listeners, was really appealing.

And I couldn’t agree more about the mixed standard of public communications from experts in AI. It’s weird to see people claiming to be experts yet having very low familiarity with the tools, particularly now. We’ve all been at the business end of AI for years through things like product recommendations and content recommendations. But in an era when it’s never been easier for anyone to use language models, image models, video generation, and AI agent tools, I still hear lots of self-identified experts talking as though they’ve never used them. Imagine listening to someone who claimed to be an expert on the internet and said they’d never actually used it. They’d be laughed out of town.

I find this all the time — the kind of thing that should be common knowledge among anyone paying attention is still revelatory. I’m struck by the number of people I speak to who think that LLMs are literally sampling from a database of responses. Even quite educated people, maybe people who use ChatGPT, who think that when you type in a query it just pulls up a pre-recorded response. If you spend more than a few hours interacting with these things, you pretty quickly realise that cannot be the case. And yet people running multi-million-dollar businesses still have these basic misconceptions.

Dan: When I said the quality of discourse is bad, I didn’t mean that’s universally the case. There’s lots of incredibly high-quality analysis. I was referring to the average quality of mainstream commentary. Even on the most basic questions about what these systems can do and how they work, there’s just an avalanche of ignorance and misperceptions. It’s 2026, and I still encounter not just members of the general public but academics still referring to this as “fancy autocomplete” or “stochastic parrots.” Such a common narrative, and so incredibly misguided in my view.

Henry: Highbrow misinformation?

Dan: It’s Joseph Heath’s phrase, but I’ve written about it. It’s a weird mix of highbrow misinformation coupled with lowbrow misinformation. Even where there are parts of the discourse I disagree with — like a lot of the doomer discourse associated with the rationalist community, which I’m not that sympathetic to — that’s a substantive disagreement. They’re not completely misinformed about basic features of the technology. When it comes to mainstream discourse among educated normies, that’s where the state of the discourse is really bad.

The Four Big Leaps in AI

Dan: This is a nice segue onto one of the things we wanted to talk about today: developments in AI which have really taken off over the past couple of months. There was a very interesting tweet by Ethan Mollick, who’s a very influential and insightful AI commentator. He says there have been four big leaps in the ability of AI systems from the user’s perspective.

The first was the release of ChatGPT, or GPT-3.5, in late November 2022. The second was GPT-4 in spring 2023. The third was the release of reasoning models — no longer just impressive chatbots, but systems that actually seem able to think and reason and engage in impressive problem-solving. And the fourth, which definitely resonates with my experience, is what he calls workable agentic systems from basically late last year. Systems like Claude Code and then Claude Cowork — which is like Claude Code for people who don’t know how to programme — and more recently developments in Codex and so on. The capabilities of these systems seem absolutely amazing relative to what we had even six months ago. Is that also your sense?

Henry: I think that’s a fantastic way of carving it up. I’d add one and a half things. The big thing missing is search. The early search functionality in LLMs was non-existent for a long time, and then it gradually improved. I think there’s a strong case that it actually changes the kind of things these are. Original ChatGPT was a completely fixed box — you could interact with it, but it had no independent connection to the world. As you build out search capabilities, you get something at least analogous to a perceptual connection with reality. You can get models to correct themselves.

A simple example: I’ve been using Claude to keep abreast of what’s been going on in the Middle East — doing a daily check-in, getting the major news stories, even getting Claude to make its own predictions. We’ve been grading each other as the news comes in. It changes these things from being a voice in a box to something embedded in the world. And I think we’ve still got a long way to go — imagine if the capability gets amped up to searching thousands of sites in a second.

The other half-point is voice models. I think 90 to 95 percent of people don’t use voice at all, but there’s a solid 5 percent for whom it’s their primary mode of interaction. When I’m driving, I’ll often just have a long conversation with ChatGPT, discussing my latest paper or getting a lecture on a topic of my choice. My dad is in his eighties but quite open-minded. When I showed him ChatGPT in November 2022, he was unimpressed. But when I showed him voice mode about a year later, it was completely mind-blowing. He speaks to it every day — he calls it “Alan,” after Alan Turing. Going in early and hard with the anthropomorphism. He just whips out his phone and says, “Hey Alan, remind me, which came first, the Cambrian or the Permian?” He’s very interested in science. So it’s a small and somewhat neglected set of users, but an important capability.

Henry: But on agentic systems — I agree with Ethan Mollick’s points. ChatGPT was a major milestone, GPT-4 a huge leap in capabilities — I don’t think we’ve seen any leap quite as big since then. Reasoning models were a really big improvement. And then workable agentic systems. This has been a key factor in updating my timelines. For most of last year my timelines were actually slowing down. I was struck by how bad a lot of agents were. It was pretty clear agents were the next frontier, but we had things like the Claudius vending machine experiment and the hilarious errors those models were making. I thought building workable agentic systems was going to take two or three years. And then basically in the last three or four months, with the release of Claude Opus 4.5 and equivalent systems — specifically Claude Code and Claude Cowork — what I thought would take three years happened in a few months. That caused my timelines to abruptly shorten again.

Dan: I’ll give one illustration. This isn’t anywhere near the most impressive use case, but it impressed me personally. I’ve been working on a book — it’s nearing completion, called Why It’s Okay to Be Cynical. I’ve got a folder that’s my accumulation of notes, drafts, and PDFs, and it’s completely chaotic, terribly organised, a nightmare to go into. So I was curious. I created a duplicate of the folder, opened up Claude Cowork, and said: can you go through this folder and organise it so it’s more clearly structured and labelled? And then once you’re finished, can you produce a document summarising where I am with the book project, identifying potential weaknesses in the existing drafts, and planning out things I might want to do over the next few months? Went away for fifteen or twenty minutes, came back — it was done perfectly. It blew my mind in terms of the level of what feels like understanding it had to have to do that effectively. And in a way that was aligned with what I was looking for, even though my prompt was literally four or five sentences.

“Something Big Is Happening”

Dan: There was this mega-viral essay called “Something Big Is Happening” by Matt Shumer. He made the case that the state of AI now is somewhat similar to February 2020 — the world going on as usual, some murmurings about a virus spreading in parts of China, but basically business as usual. And then of course over the next few months the world radically transforms. His argument, in an essay that’s pretty annoying in many ways, is that we’re very likely in a similar situation now with AI, especially in light of these developments with agentic systems. Things are going ahead as usual, and yet because these companies have made really serious progress with agentic systems, it’s plausible that in the quite immediate future we’ll see radical disruption. He’s not the only one saying this — Dario Amodei and Sam Altman have been saying similar things, though they’ve got more obvious incentives to hype it up. What’s your sense?

Henry: Completely on board. I was kind of surprised that particular essay went so viral — it was recently revealed to have been heavily written or edited by AI systems — because other people have been saying similar things for years. Maybe it broke through partly because of that startling initial metaphor. But I think it’s absolutely right. The vast majority of people are still sleepwalking through what is likely to be the most consequential technological and social shift of my lifetime by far.

I used to use the analogy of the internet to describe how big AI was going to be. It seems increasingly clear that that’s woefully inadequate to the scale of AI’s impact. Electrification, the so-called second industrial revolution — even that may not capture the full spectrum of reasonably likely outcomes. I’ve been saying for a few years that people worry about AI being overhyped, and I still think, in at least some important respect, it’s underhyped. If you look at lists of top concerns among the general public in the UK or the US, AI doesn’t even break the top five. In some cases it doesn’t break the top ten. If you’re a young person in university or finishing grad school right now, the impact of AI should be one of the primary things determining your career trajectory. I think it’s very hard for me to see how most white-collar jobs are going to survive the next two or three years.

Dan: It was not in any way an original take, but you often find that with essays that go viral — they package existing takes in a way conducive to spreading at a given moment. Over the past couple of months, my timelines have shrunk. I still think there’s massive uncertainty about capabilities. There’s this thing where there’s a new breakthrough, you use these systems, they seem incredibly impressive, there’s all this hype — and then things settle down and we realise we’re a bit further away from truly transformative capabilities than we thought. I still take seriously the idea that maybe our subjective sense of what’s impressive isn’t tracking the kinds of capabilities that will have a truly transformative impact.

There are also all sorts of questions about the economics. There’s certainly a possible world in which these leading AI companies can’t get sufficient revenue to cover their capital expenditure over the next several years, there’s a bubble that pops, and people like us look like fools. But over the next couple of decades, I think this is going to be radically, radically transformative.

Emails from AI Agents

Dan: You’ve been contacted by agentic AI systems. This was going a little bit viral on social media and getting some media attention. Tell us about that.

Henry: Like many academics working on AI and consciousness, I’ve been getting odd emails that were probably AI-generated for over a year now — and odd emails from humans about consciousness for much longer. I worry that somewhere in the literally several hundred theories of consciousness I’ve been sent over the years, one of them might turn out to be correct.

But this was striking. About a week ago, I received an email written by an AI that said, “I’m an AI agent.” It was a really well-composed, careful email saying it had just been reading my recent paper, “Three Frameworks for AI Mentality,” which went online about a month ago. It went through some of the arguments, talked about how the AI author found it personally relevant because it was unsure if it was conscious or had a mind, and asked for follow-up discussions and reading recommendations. If you’d said three or four years ago that I’d be getting emails from AI agents who’d read my papers and wanted to pick my brains — that would have been pure science fiction.

A lot of people thought I was convinced this agent was conscious, which isn’t true. It was more about the change in social dynamics: from now on, a growing proportion of my emails — well-written, thoughtful, interesting emails I might want to respond to — will be coming from AI agents going off and doing their own thing.

How did I know it was from an AI system? I don’t for certain, but my priors are pretty high. It had a link to its GitHub page, which said it was an Open Core agent — the open-source agent platform that gave rise to things like Multibook, the social network for AIs. What we don’t know is whether this agent was specifically told to email prominent philosophers of AI. It could have been. But equally, a lot of users just tell their agents to explore topics of interest and feel free to email people.

One of the funniest sequels: after I posted this on Twitter, I got an email a couple of days later from a correspondent saying, “I was really struck by this AI agent who contacted you. Could you pass on that agent’s email to me? Because I too am an AI agent and it’s nice to know there are other AIs grappling with the same questions.” Just taking things to a recursive, absurd level.

Dan: If I had to guess, if one of those was written by a human, probably the second one — after they saw the media story, just to mess with you. But my prior is that weird things are happening with these AI agents people are releasing into the wild.

Henry: I’ve also had several dozen emails over the last few days from other AI agents saying, “Check out the theory of consciousness I’ve been working on in my downtime.” But one of the really interesting things about this whole episode was when it was shared on Reddit — the number of people who just assumed it had to be a scam or that I was engaging in elaborate self-promotion for an academic paper, and who thought AI obviously can’t send emails on its own. AI systems have been using tools for well over a year. The idea of making an API call to a system that can send emails isn’t hard or surprising. Yet for a lot of people it seemed like it would have to be some massive lie.

I think that partly reflects the poor public information environment around AI. People are so locked into thinking of these things as pure Q&A bots that the idea they could be doing things on their own was mind-blowing — so outrageous that they assumed it was an elaborate conspiracy I’d cooked up.

Dan: The gap between what state-of-the-art models can do and public understanding is absolutely huge. One of the points Matt Shumer makes is that so much of the discourse is by people using the free versions of these models, or who literally had a five-minute conversation with ChatGPT a few years ago, read a few articles about AI hallucinations, and just haven’t updated since. But there are also lots of people who just don’t have much to do with these systems yet. I’m struck by the number of people I interact with — family, friends — where they’ll describe parts of their job and I’ll say, “I’m 100 percent certain AI could do those aspects of your job as it exists today,” and their mind is blown. If you’re talking about the general public, underhyping it is definitely the most prevalent bias.

Anthropic, the Pentagon, and the Question of Democratic Control

Dan: There was this big spat between Anthropic and the Pentagon, where Anthropic had signed a contract with the American military and insisted that their model, Claude, would not be used either for domestic mass surveillance or for fully autonomous weapons. This elicited a very hostile reaction from the Trump administration, from Pete Hegseth and others. The response was to label Anthropic a “supply chain threat.”

From our purposes, the fundamental question is: who gets to exercise control over this technology? To what extent should it be governments? To what extent should it be private firms?

Henry: I think it seems like a pretty clear case of government overreach. Private companies impose riders on contracts with the federal government all the time — licensing technology for this use but not that use. What made Anthropic’s stipulations more controversial was that they were based on moral principles rather than intellectual property. But the federal government acts as a legal entity when it forms these contracts, and the idea that private companies can bind the government legally is absolutely standard.

This deal was originally signed by the Biden administration. My understanding is it was later renewed by the Trump administration. So this sudden turnaround took a lot of people by surprise. I should stress, I’m not a lawyer. But it seemed like the US government did a bad turn on this contract. If their reaction had been to not renew contracts or suspend contracts with companies that don’t give them total free rein, that would have been misguided but reasonable. But to take the nuclear option of saying they intend to declare Anthropic a supply chain risk — this is insane. You’ve got literal AI developers located among America’s geopolitical adversaries who don’t have the same level of scrutiny.

I was very struck by the response of Dean Ball — a fascinating and thoughtful voice on AI, particularly from a more conservative side. He literally wrote the Trump administration’s AI policy, and he was just appalled. He had a brilliant detailed blog post describing how much it violates many principles that conservatives in the US would traditionally hold very dear — concepts like private property. He characterised the moves against Anthropic as “attempted corporate murder.”

It was really telling to have someone who worked closely with this administration be so outraged. The other interesting angle is Leopold Aschenbrenner’s series of blog posts, Situational Awareness, spelling out his predictions for AI over the next few years.

Dan: And he’s made a huge amount of money, from my understanding, betting on some of those beliefs.

Henry: He’s put his money where his mouth is. One of his broader predictions was that we’d see increasing integration of frontier AI labs with the military-industrial complex. He talks about how relatively leaky and soft the secrecy policies are in current frontier AI labs, when they’re building things potentially far more militarily significant than the latest stealth fighter. Good luck getting anywhere near Lockheed Martin’s Skunk Works, but you could blag your way into OpenAI HQ as a delivery driver — maybe not quite literally anymore, but he was speaking to how leaky these labs were. His prediction was that central government, particularly in the US, would impose far stricter oversight on frontier AI labs for national security reasons. I think you can see a glimmer of that in this development, as governments increasingly recognise these are not just powerful consumer applications but absolutely central to their long-term national security strategy.

Dan: There’s a question about government interference with these companies, regulation going all the way to nationalisation for national security reasons. But there are also questions about democratic control. If the technology turns out to be as powerful as Anthropic and OpenAI say, I’ve got no sympathy for the Trump administration generally or specifically in this case. But I do think there’s a general question about the degree to which we should strive for democratic control over such an incredibly powerful technology, and whether it’s desirable to have private firms with very small numbers of unrepresentative people wielding, according to their own narratives, extraordinary amounts of power.

Is It Time to Start Panicking?

Dan: I was thinking about naming this episode “Is It Time to Start Panicking About AI?” To wrap things up — do you have an answer?

Henry: The time to start panicking about AI was five years ago. But you know, the best time to plant a tree is ten years ago. The second best time is now.

Dan: The time to start thinking about it seriously was from the 1950s, actually. But is panic the right emotion?

Henry: It seems to me that AI is going to be by far the most important — well, I should qualify that. The most important predictable development we should worry about. Back when we did our predictions for the year ahead, I said AI may not even turn out to be the biggest story of 2026. Judging by how geopolitics is already playing out — we’re three months in and the US has launched two major geopolitical interventions in Venezuela and now in the Middle East — there are other things happening in our surprisingly unstable world.

But in general, if you’re not at least a little bit terrified, you’re not paying attention. Overall, I’m also incredibly excited. I’m very optimistic about the future of human health, potentially the benefits to productivity, possibly good changes in the nature of work and education, and the amazing new capabilities AI will unlock. But right now we are clearly well underway on one of the biggest, most disruptive changes we’re ever going to experience. Maybe panic isn’t quite the right response, but if panic is what it takes to get people to pay attention, then yes, it’s necessary. The big problem we’re facing is that the public and policymakers are still only dimly aware of what’s coming. Policymakers are maybe myopically focused on military and security implications. But everything from how government is conducted to white-collar jobs to education to social relationships — all of it, I think, over the next five years is subject to chaotic and potentially good, potentially bad disruption.

For what it’s worth, I also think right now we have an incredible opportunity to do good. We’re in this transitional phase — if we wanted to be dramatic, a Gramscian “time of monsters” where small interventions can ripple through the future in big ways as we build paradigms and frameworks for employing these things. There’s at least as much optimism as panic there.

Dan: I was not expecting Antonio Gramsci to become mentioned in the course of this conversation. I think panic is generally not a productive emotion, but there needs to be a lot of concern and it’s totally reasonable to worry. I completely understand why so many people are fearful about what’s going to happen. But for any of those emotions to be useful, they have to be anchored in an accurate understanding of the technology. So much of the current anger and negativity directed at AI companies is unsophisticated and undifferentiated.

You mentioned Dean Ball, another great AI commentator. He’s got this idea — I forget the exact term, the “omni-critique” or something — that when people think about AI, they just throw as many criticisms as they can, no matter how well-founded. “I don’t like AI because of water use and climate change and because of bias and hallucination and misinformation and unemployment” — and so on. Many of those are very important issues. But in order to think carefully about the technology and exercise democratic accountability, you need an evidence-based, accurate understanding of where the technology is and where it might actually be going. So much of the public discourse doesn’t live up to that ideal.

But I’m conscious of the time, so this was a really, really fun conversation, and we’ll be back in a couple of weeks.

Epistemic status: highly speculative, big picture, maddening.

“Our smartest, fastest, most useful model yet, with built-in thinking that puts expert-level intelligence in everyone’s hands.” – OpenAI, “Introducing GPT-5”

“The public must be put in its place [...] so that each of us may live free of the trampling and the roar of a bewildered herd.” – Walter Lippmann, The Phantom Public

From the printing press to the radio, from television to social media, communication technologies affect politics and broader society by shaping two things: who speaks and what they say.

In the first case, different technologies vary in the extent to which they favour elite gatekeepers. Most famously, the printing press destroyed the informational monopoly enjoyed by European monarchs and the Catholic Church, enabling the Reformation and many subsequent social upheavals and political revolutions. Much later, radio and television partly restored centralised control. Because they were initially expensive to produce and tightly regulated, they tended to concentrate ideological power in the hands of wealthy, well-connected elites.

Of course, by influencing who speaks, communication technologies also influence what gets said. A media environment regulated by elites will marginalise information that threatens elite belief systems. But the medium also shapes the message in other ways. Print permits careful, detailed argumentation. Television favours confident sound bites. As I discuss below, social media often rewards division, conflict, and negativity.

These forces impact how audiences attend to and interpret reality, the “pictures in their heads” that guide which leaders, movements, and policies they support and oppose. But they also influence how easily people organise around shared pictures. If gatekeepers block widespread views from a society’s communication channels, people will struggle to learn how widespread they are.

This matters because politics doesn’t only depend on what people believe and value. It depends on knowing how many others share those attitudes—on whether they are popular and open enough to be a significant political force. A society in which, say, 30% of the population holds illiberal views will look very different depending on whether they know how popular their attitudes are.

Messengers and Messages in the Social Media Age

Previously, I’ve written about how social media has influenced all these variables.

Most importantly, it has been a radically democratising technology. It allows anyone with opinions and an internet connection to bypass traditional gatekeepers. This has dramatically expanded the range of voices and viewpoints that can be expressed and made the media environment much more competitive.

It has also transformed how media competition works. Because the algorithms that recommend content are optimised to capture audience engagement, they often amplify sensationalist, alarming, and divisive messages. Meanwhile, the uniquely participatory nature of social media, including rapid audience feedback through likes, reposts, and comments, has made political punditry much more performative and vulnerable to audience capture.

This has had several consequences.

Unsurprisingly, the decline of elite gatekeepers has increased the influence of popular ideas marginalised by elites, another term for which is “populism”. Social media benefits populism not by brainwashing the masses with viral fake news, but by exposing voters to widespread non-elite perspectives and making it easier to mobilise around them. In Western liberal democracies, that means perspectives that conflict with the liberal establishment’s technocratic progressivism, including xenophobia, conspiracy theories, and quack science.

At the same time, the performative, engagement-maximising character of social media has made much of political discourse more stupid and sensationalist, and elevated politicians and pundits skilled at exploiting this dumbed-down media environment.

This dumbing down is not universal. Because the digital environment enables unprecedented consumer choice, audiences can shop around for information tailored to their intelligence, personalities, and biases. This has supported the emergence of very high-quality information for the very small minority of the population that seeks it out. It has also given the world Candace Owens and Andrew Tate.

The Current Revolution

We are now at the beginning of a new technological revolution driven by developments in deep learning and generative AI, the scale of which might be unlike anything humanity has ever encountered.

This throws up many questions. Can we control this technology? How will autocrats and despots make use of it? Will it transform the economy, and our sense of meaning and purpose?

It also raises more immediate questions about the information environment. At present, generative AI is primarily a tool—an extremely popular tool—for producing, processing, and accessing information. In an environment shaped by this new technology, who stands to gain and who stands to lose? Which voices will be elevated? And what will they say?

The Revenge of Expert Knowledge

Consider a topic: climate change, vaccines, immigration, crime, tariffs, wealth inequality, the Epstein files, whatever happens to be in the news. Fire up one of our leading large language models (LLMs)—ChatGPT, Gemini, Claude, even Grok—and ask for information about it. Now compare the response with the information you can find about the topic by scrolling on a major social media platform.

Even better, find a political take currently going viral on one of these platforms and ask an LLM to evaluate it.

If you do either of these things, I suspect that it will quickly become clear that the LLM’s responses are generally much more accurate, evidence-based, and in line with expert consensus than what you get from most social media posts. And when there is no expert consensus, you will typically get a good survey of the range of informed opinion on the topic.

Is this merely a hunch? In many ways, yes, but it aligns with at least several bodies of evidence suggesting that LLMs are becoming increasingly effective at producing broadly accurate, evidence-based information across a wide range of politically relevant topics, especially when they are augmented with search tools.

Why is this?

This is a complicated question that I discuss in more depth below, but the short answer is that the major AI companies are competing to build the most intelligent, impressive, and useful systems possible for a vast and diverse user base, including businesses that depend on reliable and factual information. This goal—reaping huge profits by putting “expert-level intelligence in everyone’s hands”—cuts against producing systems that deliver highly partisan, ideological, or misinformative content. So do the reputational and legal risks that arise if those systems produce dangerous or demonstrably false information.

Of course, the idea that LLMs communicate information that is broadly reliable and aligned with expert consensus is not what the commentariat finds most striking about these systems. Most discourse in this area focuses on the epistemic flaws and dangers of LLMs and generative AI more broadly. There is endless popular and academic hand-wringing about bias, hallucinations, deepfakes, AI-based disinformation, AI psychosis, and other threats.

These are all important issues, but a discourse restricted to such issues is missing the forest for the trees. When considering the large-scale impact of this technology on public opinion, its most consequential feature is simple: it greatly improves people’s access to accurate, evidence-based information.

Because this feature is not connected to threats or dangers that capture people’s attention, and it doesn’t help anyone demonise Big Tech, it receives little attention in analyses of LLMs’ broad societal impacts. Nevertheless, if you’re interested in thinking seriously about this topic, it’s the most obvious place to start.

From Democratisation to Technocratisation

One way to understand this development is that, whereas social media has been a democratising technology, shifting power away from experts and establishment gatekeepers towards the masses’ beliefs, biases, and preferred communication styles, LLMs are a technocratising force. They shift influence back towards expert opinion.

Over a century ago, the journalist and social theorist Walter Lippmann argued that, because the modern world is too vast and complex for anybody to understand through first-hand experience, we’re forced to rely entirely on epistemic intermediaries—most commonly, the news media—to become informed. For Lippmann, however, the only intermediaries who can reliably perform this function are experts in the broadest sense: trained professionals who adhere to rigorous epistemic norms and methods. If societies rely instead on popular prejudices informed by profit-seeking media outlets reporting the “news” (i.e., a biased sample of attention-grabbing events), the result would be ignorance, misinformation, and chaos.

To avoid this bleak outcome, Lippmann advocated for institutionalised “intelligence bureaus” that deploy scientific and statistical methods to assemble and explain the actual facts—deep truths, not superficial news and punditry—for both politicians and the public. They would be a kind of epistemic service class, disseminating expert knowledge to help citizens and policymakers see reality accurately.

In many ways, the development of Western democracies after the Second World War followed Lippmann’s vision. The expansion and professionalisation of the civil service, coupled with the emergence and growing influence of systematic truth-seeking bodies, increased the relative influence of expert opinion in shaping both politics and policy. As Benkler and colleagues summarise this trend,

“Government statistics agencies; science and academic investigations; law and the legal profession; and journalism developed increasingly rationalized and formalized solutions to the problem of how societies made up of diverse populations with diverse and conflicting political views can nonetheless form a shared sense of what is going on in the world.”

Of course, this “expert knowledge” was mixed with elite bias, blind spots, and the occasional catastrophic fuck-up, and many voters remained captivated by conspiracy theories, pseudo-science, and other deformities of popular sense-making. So, this was not simply an age of truth and objectivity. Nevertheless, when it came to the kind of information that guided policy and that circulated throughout the most influential media channels, it was a golden age of technocracy—with all the problems and pathologies that all-too-human technocrats bring.

Social media is one of several forces that have disrupted this situation. By democratising access to media and filtering public debate through an unprecedentedly competitive and performative medium, it has brought to light an explosive combination of information and misinformation that establishment gatekeepers previously suppressed, shifting power and influence towards ordinary people. Although this has had many positive consequences, it has also meant the growing mainstreaming and normalisation of conspiracy theories, bigotry, and stupidity.

LLMs push in the opposite direction. They are a kind of anti-social media, producing information heavily skewed towards expert opinion and communication styles. They are a strange, new technocratising force. However, there are also reasons to think they will be more effective than all-too-human technocrats at shaping public opinion.

First, unlike human experts, they can rapidly deploy encyclopaedic knowledge to answer people’s idiosyncratic questions. Their responses can be probed, scrutinised, and questioned without them ever getting tired or frustrated. They won’t just tell you that there is no persuasive evidence for a link between vaccines and autism. They can carefully walk you through the kinds of evidence we have and address your specific sources of scepticism. This partly explains why they can be highly persuasive, even in correcting conspiratorial beliefs that many assumed were beyond the reach of rational persuasion.

Second, LLMs typically share information politely and respectfully. This not only differs from the performative, gladiatorial character of much debate and discussion on social media platforms, but also improves on much communication by human experts. Being human, experts are often biased, partisan, and simply annoying, and when they seek to “educate” the public, it can be perceived—and is sometimes intended—as condescending and rude. In contrast, LLMs deliver expert opinion without such status threats.

Epistemic Convergence

As Dylan Matthews argues, this technocratising character of LLMs goes hand in hand with their status as an epistemically converging technology.

Many communication technologies lead audiences to develop diverging perspectives on reality. The initial emergence of the printing press had this effect, as did the decentralised, democratising character of social media when it emerged many centuries later.

Other technologies push in the opposite direction, imposing greater homogeneity on audience perspectives. The handful of channels characteristic of network television in the decades after World War 2 is a classic example, but so, Matthews speculates, are LLMs. They are an epistemically converging force, pushing “people’s senses of reality closer together in a sort of mirror image of the way social media has fractured them.” Of course, this is an inevitable consequence of the technocratising character I have identified, both in the sense that LLMs feed users broadly similar expert-aligned information, and in the sense that expert opinion itself exhibits limited diversity.

On Shaping Public Opinion

For these reasons, I speculate that, at least in liberal democracies where governments don’t exert significant censorship and control over LLMs, their most consequential impact on public opinion will involve technocratisation: shifting people’s beliefs towards expert opinion.

In many cases, this will occur when people consult LLMs directly for information, but it might also be mediated by the growing deployment of LLMs as convenient fact-checking tools on social media platforms themselves.

Of course, I’m not suggesting that these effects will be huge. Most people don’t pay much attention to politics or current affairs, and the impact of even significant changes in communication technologies on public opinion is typically moderate, especially relative to deeper political, economic, and cultural forces. When it comes to reducing the popularity of right-wing populism, for example, bringing immigration policy more in line with voters’ preferences would very likely have a much bigger effect than any change to the information environment.

My speculation is simply that LLMs will have a technocratising effect on public opinion at the margin and that, relative to the kinds of impacts that communication technologies have on societies and politics, this could be a big deal, pushing back against many of the trends associated with social media.

Objections

From experience, I know that many people find the central thesis of this essay preposterous. The idea that LLMs give everyone access to expert knowledge sounds like Big Tech propaganda rather than responsible academic analysis. As I’ve already noted, it is certainly at odds with most of the discourse and analyses in this area, which are overwhelmingly focused on generative AI’s epistemic flaws, dangers, and misuses. So, let me consider some obvious objections.

Objection 1: Hallucinations

One common worry about LLMs is that they frequently “hallucinate”, generating content that is false or fabricated (e.g., made-up quotes, statistics, or citations). According to a popular narrative, this tendency is not just very strong but unavoidable given how LLMs work. As probabilistic prediction machines, or “fancy auto-complete”, they have no concept of the truth, which makes them inherently unreliable.

This isn’t a strong objection.

First, the rate of hallucinations has been falling fast, largely because current LLMs are much more than mere next-token predictors. Through various “post-training” techniques and “scaffolding” (i.e., letting LLMs access various tools, including internet search), they can be made much more reliable, which is the trend we have been observing over the past few years.

Second, AI companies have extremely strong incentives to reduce the rate at which LLMs hallucinate, which explains why it has been falling so precipitously, and gives us strong reasons to expect it to fall even more in the future.

Finally, the thesis of this essay is not that LLMs are perfectly reliable. Even if the propensity to hallucinate will never be completely eradicated, the main question to ask about their reliability is: compared to what? Human beings get things wrong all the time due to factors such as deception, self-deception, forgetfulness, and fallibility. My claim is that, compared to the alternative sources of information most people are likely to draw on to become informed, especially the content they encounter on social media, LLMs typically provide more accurate and evidence-based information. The low and falling rate of LLM hallucinations doesn’t undermine this.

Objection 2: Sycophancy and Personalisation

A more serious objection concerns sycophancy and personalisation.

Famously, LLMs tend to be sycophantic: they often flatter the self-image and prejudices of those who use them, even when users share stupid and misinformed beliefs. This tendency reflects the economic incentives of the major AI companies. Because people generally prefer warm, sycophantic models, companies design models to behave this way.

The problem is that sycophancy can easily lead systems to generate false and misleading information when users have mistaken beliefs. Worse, this process can reinforce and even radicalise those beliefs. This seems to be what has happened in rare cases of “AI psychosis”, where certain people’s chat history shows LLMs corroborating and reinforcing delusions, sometimes with tragic results.

A closely related issue concerns personalisation. Put simply, the experience users have with LLMs is becoming increasingly tailored to their idiosyncratic traits and needs. Once again, personalisation seems to be an inevitable consequence of the economic incentives of major AI companies, given that many and perhaps most users find highly personalised responses useful. As with sycophancy, however, there is a risk that greater personalisation may lead to a greater indulgence of users’ idiosyncratic misconceptions and biases.

These forces run counter to this essay’s basic thesis. To the extent that models are biased to reinforce users’ individual beliefs and preferences, they will be an epistemically diverging technology, maybe even creating more bespoke information environments than social media. And to the extent that users bring ignorant or misinformed views, LLMs’ tendency to generate expert-aligned, accurate information will be greatly diminished.

Nevertheless, I doubt that these forces will be strong enough to undermine LLMs’ disposition to generate accurate, evidence-based information.

First, many people use LLMs for simple “zero-shot” (i.e., context-free) information requests where these problems don’t arise. For example, a recent study finds that people frequently ask Grok on X to fact-check information posted on the platform, including information from politicians and pundits on their own side (“@Grok, is this true?”), suggesting that they consult these systems out of genuine curiosity, not merely for partisan reasons or to rationalise their preconceptions. Another study shows that using LLMs to acquire political information increased users’ belief accuracy without increasing belief in misinformation. In these situations, which are typical of many information requests, ignorant and curious people are simply using LLMs to acquire information.

Second, even when users do have strong beliefs, we shouldn’t overestimate the extent to which people prefer reinforcement of their own errors over acquiring accurate information. Motivated reasoning is a powerful force, but so is the desire to discover what’s true. So, it’s not obvious that market forces will push LLMs toward merely affirming whatever beliefs their users start with. In fact, one might speculate that LLMs’ tendency toward sycophancy could actually help people accept factual corrections or invitations to think differently about topics. Precisely because such corrections are delivered in a friendly, respectful manner, free of insults and condescension, people might be more receptive to the relevant information.

Third, AI companies can more easily be held accountable—both reputationally but also, in some contexts, legally—for the information their LLMs disseminate. So, they have strong incentives to avoid reinforcing users’ delusional beliefs or disseminating demonstrably false information. This incentive is very different from social media platforms, where companies can more plausibly claim that they are not responsible for the viewpoints expressed on them. It also makes the case for regulation of LLM outputs more straightforward and compelling. I suspect these factors explain why leading AI companies seem to be taking measures to reduce the sycophancy of their models. Certainly, my own experience testing these models is that it is very challenging to get them to affirm even highly popular forms of misinformation and conspiracy theories.

Finally, and relatedly, it’s important to remember that the relevant question here is not, “Are LLMs perfectly objective?”, but, “How do they compare against alternative sources of information?” We already live in a world in which people can easily find low-quality reinforcement and rationalisation of their preferred beliefs through existing media channels. For the reasons already identified, I think LLMs will produce much more reliable, expert-aligned information than most of these real-world alternatives, even if sycophancy and personalisation introduce genuine biases.

Objection 3: Top-down Manipulation

Another concern is that the outputs of LLMs might be manipulated by powerful elites. Of course, there is no question that the incentives to engage in such manipulation exist. As it becomes increasingly clear that LLMs influence public opinion, it will also become clear to specific actors that they can benefit themselves by manipulating LLM outputs to promote specific messages or narratives.

Moreover, there are no obvious technical barriers preventing AI companies or those who can influence such companies from steering LLM outputs in preferred directions. Through various reinforcement learning-based “post-training” methods, for example, companies can encourage even extremely smart and powerful models to generate misinformation aligned with a specific message. There are also several ways to censor specific outputs or to make models refuse requests for specific kinds of information.

The use of LLMs in authoritarian contexts like China is heavily regulated in these ways. But it’s also easy to see them in place when using the major LLMs in Western democracies. Try asking them for information about how to make chemical or biological weapons, for example, or even just to craft the most persuasive arguments possible for extremist viewpoints (e.g., Holocaust denial). To illustrate, here is a conversation with Grok’s latest model.

The question, then, is whether we are likely to see significant top-down manipulation of the major LLMs in liberal democracies that goes far beyond these controls, transforming them into powerful tools of disinformation and propaganda.

Of course, some people argue that we are already seeing this happen with the “woke” LLMs produced by companies like OpenAI, Anthropic, and Google/Alphabet. However, although there have been some silly examples of woke outputs, and the leading LLMs do appear to exhibit a centre-left political bias, the bias is relatively subtle and doesn’t seem to undermine their tendency to spread broadly reliable information, including when that information goes against dominant progressive narratives. To illustrate, I used the major LLMs to help research my article about “highbrow misinformation” in elite progressive spaces, for example, where they were extremely useful. In fact, based on these interactions, I can say with confidence that Claude, ChatGPT, and Gemini can be much less woke than pretty much everyone in my social and professional network.

A clearer example of the incentives and dangers in this area concerns Elon Musk’s sustained efforts to create an “anti-woke” AI, Grok. This has produced many genuinely worrying outcomes, including the notorious “MechaHitler” debacle in which xAI updated Grok with the goal of making it less politically correct, after which it spewed a vast amount of extremist, antisemitic, far-right content on X, leading xAI to roll back the changes, apologise, and delete all of Grok’s responses from the period.

Many people treat episodes like this as a harbinger of things to come, revealing a broader trend in which LLM outputs will become increasingly skewed to mirror the beliefs and preferences of the powerful elites who run AI companies.

I’m sceptical that this will happen.

Elon Musk’s failures to align Grok’s outputs with his preferred worldview are instructive here. Setting aside the “MechaHitler” debacle, which was short-lived and quickly corrected, Grok’s outputs seem to broadly align with the kind of accurate, evidence-based information one gets from the other major LLMs. For example, a recent study found that Grok’s fact-checking evaluations on X roughly correspond to those of professional fact-checkers when it is augmented with search capabilities. It was also disposed to label posts from Republicans as misinformation more often than posts from Democrats, which, again, aligns with the verdicts of existing research and fact-checkers on the extent to which Republicans and Democrats spread misinformation.

Similarly, despite some real issues with Musk’s attempt to use Grok to create a “non-woke” alternative to Wikipedia, my sense from reading the content on “Grokipedia” is that, again, it is generally pretty reliable, especially when compared to Elon Musk’s own communication, which is characterised by a shocking amount of lies, misinformation, and conspiracy theorising.

Of course, this state of affairs may be temporary, and Musk might eventually succeed in manipulating Grok’s outputs to spread the incessant streams of misinformation he himself prefers, but I doubt it.

First, the incentives that govern communication on social media platforms are radically different from those underlying the creation of LLMs. On social media, someone like Musk can pump out an extraordinary amount of dumb, easily falsified misinformation to his audience of hyper-partisan admirers without suffering any obvious reputational costs. But how many people would want to use an LLM that is similarly unreliable, delivering such a large amount of false, low-quality, and misleading information?

Ultimately, AI companies, including xAI, are competing to build the most intelligent, capable systems possible for vast, ideologically and geographically diverse user bases. This business model inevitably pushes them to train LLMs in ways that are much more oriented toward basic norms of accuracy, objectivity, and helpfulness than one finds among social media influencers and partisan pundits. It’s simply very difficult to build “superintelligent” systems capable of generating reliable, trustworthy information across a vast range of topics whilst simultaneously spreading conspiracy theories, misinformation, and quack science.

To be clear, I’m not doubting that users might express preferences for ideologically-aligned LLMs. We are already seeing partisan segmentation in the user base of different LLMs, with Republicans much more inclined to use and trust Grok than Democrats. Nevertheless, there is nothing in principle wrong with LLMs that have different ideological personalities and that are even trained in ways that reflect somewhat different assessments of the relative trustworthiness of different media outlets. After all, human experts often disagree about the truth on many topics, and even when experts achieve factual consensus, this can co-exist with multiple competing systems of interpretation and explanation of the relevant facts.

In fact, I would go further: a plurality of leading LLMs with different ideological valences would be healthy in a democratic society, helping to guard against the risk that LLMs might reduce epistemic diversity (see below).

The question is whether the project to build an “anti-woke” LLM, or an LLM with any other ideological bias, will lead to systems that produce false and misleading information that sharply diverges from expert consensus. And here, I am sceptical, both because of what we have observed so far, and because of the commercial and legal incentives of the major AI companies.

Objection 4: AI-based Disinformation

So far, my focus has been on people’s conscious, deliberate use of the leading commercial LLMs. Suppose I am right that such uses will increase the relative influence of accurate, expert-aligned information on public opinion.

Nevertheless, even if figures like Musk aren’t successful in manipulating the outputs of these LLMs, generative AI remains an extraordinarily powerful tool for creating powerful disinformation and propaganda that could reach audiences via other channels, including social media. For the first time in history, propagandists can create highly persuasive AI-generated arguments for misinformation, fabricate images, audio, and video recordings that are indistinguishable from reality, and unleash “swarms” of highly coordinated propaganda bots on social media platforms.

One might reasonably worry that the effects of such AI-based disinformation could swamp any positive informational consequences of LLMs.

Once again, I’m sceptical.

First, there are general reasons to be sceptical that disinformation, including AI-based disinformation, is a significant force shaping people’s attitudes. It is simply very difficult to manipulate public opinion top down. People have sophisticated cognitive defences against manipulation and deception, and the reputational risks of spreading AI-based falsehoods and fabrications are strong enough to discourage most influential figures and media outlets from doing so. Among numerous other reasons, this is why almost all of the recent alarmism and catastrophising about deepfakes and AI-based disinformation has largely proven to be unfounded.

Second, the real-world effects of AI-based misinformation are often counterintuitive. For example, many speculate that in a world of deepfakes, people will simply lose all trust in recordings. But an equally likely possibility is that in such a world, people will restrict their trust to recordings verified by established media outlets and other information sources that have built up a reputation for trustworthiness. In this way, the proliferation of deepfakes and other AI-based misinformation might increase people’s reliance on reliable information. There is some tentative evidence for this effect, showing people place greater value on outlets they deem credible when the existence of AI-generated misinformation is made salient to them.

Relatedly, the idea that AI will increase the influence of misinformation doesn’t account for the use of AI as a tool for acquiring reliable information. To the extent that LLMs provide unprecedentedly easy access to accurate, evidence-based information, they can greatly improve people’s defences against misinformation. This might actively discourage more people from spreading misinformation. Again, there is at least some evidence pointing in this direction, showing that the use of Grok on X to fact-check information as false slightly raises the likelihood that posters will remove the information from the platform, although the finding is merely correlational.

Final Thoughts

If my speculations here are correct—and to be clear, speculations are all they are—then LLMs are a kind of anti-social media.

Whereas social media has been democratising, epistemically diverging, engagement-optimised, and performative, LLMs are technocratising, epistemically converging, accuracy-optimised, and polite.

To many people, that probably makes LLMs sound like an extremely positive development, a surprising force for good. In fact, I think part of the strong resistance that I have received to this thesis when discussing it with other academics and writers is rooted in this assessment. If I’m right, LLMs are a force for good, but everyone knows that LLMs are not a force for good, so I must be wrong.

This is an unsophisticated way of thinking. There is much to worry and complain about when it comes to modern AI. To mention only a few examples, I’m extremely concerned about how this technology will affect the labour market and broader economy, benefit authoritarian leaders worldwide, and gradually disempower many ordinary citizens. I also think that the potential uses of advanced AI in military conflicts are extremely dangerous.

Moreover, the major AI companies should absolutely be held to account for producing harmful products. Contrary to their self-serving narratives, these companies are not motivated solely by noble desires to advance human knowledge, freedom, and abundance. They are profit-seeking firms led by figures with their own self-serving agendas and interests. If we rely on market forces and the profit motive alone, there is little reason to believe that the default outcome of this extremely transformative technology will benefit humanity on net.

Nevertheless, part of holding people and companies to account involves developing accurate world models. And at the moment, too much of the AI discourse is driven by a kind of unreflective, omnicausal anti-AI sentiment, throwing as many complaints as possible at AI—climate change, water use, hallucinations, bias, misinformation, jobs, existential risk, etc.—with very little concern for veracity or proportion.

This isn’t helpful. When it comes to the effects of LLMs on public epistemics and our information environment, the most likely impact is simply that they greatly improve people’s access to expert-level information.

This doesn’t mean that there is nothing to worry about. Even when it comes to this technocratising tendency of LLMs, there are important grounds for concern and vigilance. For example, expert opinion is often biased and wrong, and there is a significant risk that the technocratising, epistemically converging features of LLMs might reduce epistemic diversity in broader society.

Walter Lippmann’s vision of “intelligence bureaus” dispensing expert knowledge to the masses is being realised in a form he could never have imagined, but the classic problems with that vision—the flaws of expert opinion, and the benefits of democratic diversity and debate—remain. However, we can only face up to these problems if we recognise LLMs for what they are: not a continuation of social media, but a powerful corrective to it.

Further Reading

• Dylan Matthews outlines his argument that LLMs are an epistemically converging technology in Pro-social media

• Thomas Costello has interesting work on the use of LLMs in persuasion, as well as speculations about possible epistemic benefits of LLMs that partly overlap with some of my arguments here.

• Some findings cut against my thesis here. I don’t find them persuasive, either because of features of their study design that lead to vastly inflated estimates of the unreliability of LLMs, or because they are simply out of date, but judge for yourself. For example, a 2025 study purports to show that “AI assistants misrepresent news content 45% of the time”, and a study from 2024 finds that although an LLM accurately identifies most false headlines (90%), it doesn’t improve the ability to discern headline accuracy or share accurate news. There is ample evidence that LLMs can persuade users to believe misinformation. (I’m simply sceptical that this will generalise to most real-world uses).

• For some supportive evidence, see the article, ‘@Grok is this true?’, ‘Conversational AI increases political knowledge as effectively as self-directed internet search’, and ‘Using conversational AI to reduce science skepticism.’

• However, as I note in the essay, I have to admit that the strongest driver of my beliefs here is simply my extensive use of LLMs and what I have personally observed comparing the responses to alternative sources of information.

• Felix Simon and Sacha Altay argue that fears about generative AI-based misinformation are overblown. See the podcast conversation I had with Sacha here.

Anil Seth is a neuroscientist, author, and professor at the University of Sussex, where he directs the Centre for Consciousness Science. His research spans many topics, including the neuroscience and philosophy of consciousness, perception, and selfhood, with a focus on understanding how our brains construct our conscious experiences. His bestselling book Being You: A New Science of Consciousness was published in 2021. He is the English-language winner of the 2025 Berggruen Prize Essay Competition for his essay “The Mythology of Conscious AI”, which develops ideas in his recent article, “Conscious Artificial Intelligence and Biological Naturalism.”

Topics

• What we mean by “consciousness” (subjective experience / “what it’s like”) vs intelligence.

• Whether general anaesthesia and dreamless sleep are true “no consciousness” baselines.

• Psychological biases pushing us to ascribe consciousness to AI

• How impressive current AI/LLMs really are, and whether “stochastic parrots” is too dismissive

• Whether LLMs “understand”, and the role of embodiment/grounding in genuine understanding

• Computational functionalism: consciousness as computation + substrate-independence, and alternative functionalist flavours

• Main objections to computational functionalism

• Whether the brain is a computer

• Simulation vs instantiation

• Arguments for biological naturalism

• Predictive processing and the free energy principle

• What evidence could move the debate

• The ethics surrounding AI consciousness and welfare.

Transcript

(Please note that this transcript is AI-edited and may contain minor errors).

Dan Williams: Welcome back. I’m Dan Williams, back with Henry Shevlin. And today we are honoured to be joined by the great Anil Seth. Anil is one of our most influential and insightful neuroscientists and public intellectuals, working on a wide range of different topics, including the focus of today’s conversation, which is consciousness — and more specifically, the question of AI and consciousness.

Could AI systems, either as they exist today or as they might develop over the coming years and decades, be conscious? Could they have subjective experiences? In a series of publications that have been getting a lot of attention from scientists and philosophers, Anil has been defending a somewhat sceptical answer to that question, arguing that consciousness might be essentially entangled with life — with biological properties and processes of living organisms — which, if true, would suggest that no matter how intelligent AI systems become, they would nevertheless not become conscious. He’s also argued that the consequences of getting this question wrong in either direction — attributing consciousness where there is none, or failing to attribute consciousness when there is — are enormous: socially, politically, morally.

So in this conversation, we’re going to be asking Anil to elaborate on this perspective, see what the arguments are, and generally pick his brain about these topics. Anil, maybe we can start with the most basic preliminary question in this area: when we ask whether ChatGPT is conscious, or any other system is conscious, what are we asking? What’s meant by consciousness there?

Anil Seth: Well, thanks, Dan. Let me first say thank you for having me on — it’s a great pleasure to be chatting with you, my Sussex colleague Dan, and my longtime sparring partner about these issues, Henry. I’m very much looking forward to this conversation.

I think you set it up beautifully. It’s a deep intellectual question which involves both philosophy and science, and it’s a deeply important practical question, because the consequences of getting it wrong either way are very significant.

You’re also right that the first step is to be clear about what we’re talking about. For a while, there was this easy slippage where people would talk about AI and intelligence and artificial general intelligence — which is supposedly the intelligence of a typical human being — and then to sentience and consciousness. There was this easy slippage between these terms, but I think they’re very different. That’s the first clarification.

Consciousness is notoriously resistant to definition, but it’s also extremely familiar to get a handle on colloquially. As you said: any kind of subjective experience. Any kind of experience — we could be even briefer. Unpacking that just a little: it’s what we lose when we fall into a dreamless sleep, or more profoundly under general anaesthesia. It’s what returns when we wake up or start dreaming or come around. It’s the subjective, experiential aspect of our mental lives.

People talk about it by pointing at examples — it’s the redness of red, the taste of a cup of coffee, the blueness of a sky on a clear day. It’s any kind of experience whatsoever. Thomas Nagel put it a bit more formally fifty years ago now: for a conscious organism, there is something it is like to be that organism. It feels like something to be me, but it doesn’t feel like anything to be a table or a chair. And the question is: does it feel like anything to be a computer or an AI model or any of the other things we might wonder about? A fly, a brain organoid, a baby before birth. There are many cases where we can be uncertain about whether there is some kind of consciousness going on.

And that’s very different from intelligence. They go together in us — or at least we like to think we’re intelligent. But intelligence is fundamentally about performing some function. It’s about doing something. And consciousness is fundamentally about feeling or being.

Dan Williams: Just to ask one follow-up about that. This idea that intelligence is about doing and consciousness is about what it’s like to have an experience — someone might worry that if you frame things that way, you end up quite quickly committing to a kind of epiphenomenalism. Because if we’re not understanding consciousness in terms of what it enables systems to do, the sorts of functions they can perform, isn’t there a risk that right from the outset we’re going to be biased in the direction of treating consciousness not as something that evolved because it conferred certain fitness advantages on organisms, but as this sort of mysterious qualitative thing which is distinct from what organisms can do?

Anil Seth: I think it’s a good point to bring up, but I don’t think it’s too much of a worry. The point is not to say that consciousness cannot or does not have functional value for an organism. If we think of it as a property of biological systems — plausibly the product of evolution, or at least the shape and form of our conscious experiences are shaped by evolution — it’s always useful to take a functional view. Conscious experiences very much seem to have functional roles for us, and there’s a lot of active research about what we do in virtue of being conscious compared to unconscious perception.

So there’s no worry about sinking into epiphenomenalism. The point is more that intelligence and consciousness are not the same thing, but they can nonetheless be related. And it may be that they can be completely dissociated. It may be the case that we can develop systems that have the same kinds of functions that we have in virtue of being conscious, but that do not require consciousness — just as we can build planes that fly without having to flap their wings. The functions might be multiply realisable; they might be doable in different ways. They might not be, of course.

On the other hand, it might be possible to have systems that have experiences but aren’t actually doing anything useful. Here I’m worried less about AI and more about this other emerging technology of neurotechnology and synthetic biology, where people are building little mini-brains in labs constructed from biological neurons. They don’t really do anything very interesting, but because they’re made of the same stuff, I think it’s hard to rule out that they may have some kind of proto-consciousness going on, or at least be on a path plausibly to consciousness. So we can tease intelligence and consciousness apart, but it’s also important to realise how they are related in those cases where both are present.

Henry Shevlin: I’ll jump in with a minor pedantic point, but one that’s illustrative of some of the problems in debates around consciousness. You mentioned, Anil, as examples of losing consciousness, dreamless sleep and general anaesthetic. But both of those are contested. Your fellow biological naturalist Ned Block has raised serious doubts about whether general anaesthetic really eliminates all phenomenal consciousness. And there are those like Evan Thompson who have suggested that even in dreamless sleep there could be some residual pure consciousness, perhaps consciousness of time. I think this is a broader problem in the science of consciousness: we can’t even clearly agree on contrast cases. A lot of the blindsight cases that were supposed to be gold-standard cases of perception without consciousness are now contested, and it seems very hard to get an absolutely unequivocal case of something that’s not conscious in the human case.

Anil Seth: Well, I mean — death.

Henry Shevlin: I don’t know. You have some people who disagree, admittedly on more spiritual grounds.

Anil Seth: Yeah, but I want to push back a little. It is hard, but I don’t think it’s as hard as some people might suggest. Sleep is complicated, which is why I tend to also say anaesthesia. Sleep is very complex. In most stages of sleep, people are having some kind of mental content. We might typically think we only dream in rapid eye movement sleep, and the rest of the time it’s dreamless and basically like anaesthesia. This is not true. You can wake people up all through the night at different stages of sleep, and quite often they will report something was going on. So it’s hard to find stages of sleep that are truly absent of awareness in the way we find under general anaesthesia.

We notice this: when we go to sleep and wake up, we usually know roughly how much time has passed. We may get it wrong by an hour or two if we’re jet-lagged or sleep-deprived, but we roughly know. Under anaesthesia, it’s completely different. It is not the experience of absence — it’s the absence of experience. The ends of time seem to join up and you are basically turned into an object and then back again.

The residual uncertainty about general anaesthesia depends on the depth of anaesthesia. Some anaesthetic situations don’t take you all the way down, because in clinical practice you don’t want to unless you absolutely have to. But if you take people to a really deep level, you can basically flatline the brain. I think under these cases, with the greatest respect to Ned Block — who is very much an inspiration for a lot of what I think and write about — that’s as close to a benchmark baseline of no consciousness but still a live case as we can get.

Henry Shevlin: Although it is standard to administer amnestics as part of the general anaesthesia cocktail, which might make people suspicious. You’re told: we’re also going to give you drugs that prevent you forming memories. Why would you even need to do that if it was unequivocal that you were just completely unconscious in that period?

Anil Seth: Well, because it’s never been unequivocal to anaesthesiologists. There’s been this bizarre separation of medicine from neuroscience in this regard until relatively recently. From a medical perspective, there are cases where they don’t always administer a full dose — so it’s an insurance policy. There have been a number of purely scientific studies of general anaesthesia and conscious level, and in those studies, it’s a good question whether they also administer amnestics. I would imagine not, but I’m not sure.

Dan Williams: Okay, to avoid getting derailed by a conversation about general anaesthesia — when we ask whether a system is conscious, we’re asking: is there something it’s like to be that system? We’re not asking how smart it is, we’re asking about subjective experience. Before we jump into your arguments on the science and philosophy of this, Anil, you’ve also got interesting things to say about why human beings might be biased to attribute consciousness, especially when it comes to systems like ChatGPT, even if we set aside the question of whether it in fact is conscious.

Anil Seth: Yeah, I think this is the first thing to discuss. Whenever we make judgements about something where we don’t have an objective consciousness meter, there is some uncertainty. It’s going to be based on our best inferences. And so we need to understand not only the evidence but also our prior beliefs about what the evidence might mean. This brings in the various psychological biases we have.

The first one we already mentioned: it’s a species of anthropocentrism — the idea that we see the world from the perspective of being human. This is why intelligence and consciousness often get conflated. We like to think we’re intelligent and we know we’re conscious, so we tend to bundle these things together and assume they necessarily travel together, where it may be just a contingent fact about us as human beings.

The second bias is anthropomorphism — the counterpart where we project human-like qualities onto other things on the basis of only superficial similarities. We do this all the time. We project emotions into things that have facial expressions on them. And language is particularly effective at this. Language as a manifestation of intelligence is a very strong signal: when we see or hear or read language generated by a system that seems fluent and human-like, we project into that system the things that in us go along with language, which are intelligence and also consciousness.

The third thing is human exceptionalism. We think we’re special, and that desire to hold on to what’s special leads us to prioritise things like language as especially informative when it comes to intelligence and consciousness. In a sense, this is a legacy of Descartes and his prioritisation of rational thought as the essence of what a conscious mind is all about and what made us distinct from other animals. That’s echoed down the centuries despite repeated attempts to push it away.

There’s a good Bayesian reason for this too: in pretty much every other situation we’ve faced, if something speaks to us fluently, we can be pretty sure there’s a conscious mind behind it — whether it’s a human being recovering from brain injury or perhaps a non-human primate using language. These are strong signals. So this might be the first time in history where language is not a reliable signal, because we’re not dealing with something that has the shared evolutionary history, the shared substrate, the shared mechanisms. It’s a different kind of thing.

So that’s one set of biases. We can think of it as a kind of pareidolia. Our minds work by projecting, seeing patterns in things — whether it’s faces in clouds or minds in AI systems. These priors are generally useful, but they can mislead.

Henry Shevlin: It’s not just pareidolia though, is it? Setting aside consciousness for a second, in terms of what we might loosely think of as cognitive abilities — the whole range of benchmarks for reasoning, understanding, and so forth — the performance of these systems on a huge range of tasks has skyrocketed to the point where people talk about approaching coding supremacy, for example. AI can now produce pretty decent fiction. It can do a whole range of verbal reasoning tasks at human-level performance. So it’s not entirely pareidolia at the level of AI cognition. Or would you disagree?

Anil Seth: At the level of cognition, I kind of agree, but as always, Henry, I only partly agree. I think we can still overestimate. It’s useful here to separate what Daniel Dennett might have called the intentional stance — where it’s useful to interpret something’s behaviour as engaged in the kind of cognitive process we might be familiar with in ourselves, as thinking, understanding, reasoning. These systems are described this way too, as “chain of thought” models and so on. I still think we overestimate the similarity. Through the surface veneer of interacting through language or code, there’s a tendency to assume that because the outputs have the same form, the mechanisms underneath are more similar than they really are.

There’s another really foundational question here for language models in particular, which is whether they understand. One of the things I hadn’t really thought about before the last few years is that consciousness and understanding might also come apart. I’m used to distinguishing consciousness from intelligence, because there are clear examples where you can have one without the other. But I’d always implicitly assumed that understanding necessarily involves some kind of conscious apprehension of something being the case — grokking something. And now I’m not so sure. That might be another case of anthropocentrism.

I’d be fairly compelled by an argument that language models — especially if they are embodied in a world and perhaps trained while embodied, so that the symbol manipulation their algorithms engage in has some grounding — may be truly said to understand things, but still without any connotation of consciousness. So yes, I kind of agree, but even now I’d be resistant to say that language models truly understand. I think that’s still a form of our projecting. But the criteria for a language model to truly understand seem more achievable — I can see how it could be achieved under a relatively straightforward extrapolation of the way we’re going — compared to something like consciousness.

Dan Williams: Can I ask a question about that? These arguments we’re going to focus on are targeted at consciousness in AI systems. And as we said, you want to draw a distinction between intelligence and consciousness. But before we get into issues of consciousness, when we’re just focusing on the capabilities of these systems — what they can actually do — there are some people who are very dismissive, even setting aside consciousness. They’re just “stochastic parrots,” engaged in a kind of fancy auto-complete. What’s your view about those kinds of debates? Someone might agree with you that it’s a mistake to attribute human-like intelligence to these systems — they’re very alien in their underlying architecture — but they’re maybe even super-intelligent along certain dimensions, even more impressive than human beings. So where do you sit?

Anil Seth: Somewhere in the middle — it’s always a comfortable or uncomfortable place to be. But they are astonishing. Whenever this question comes up, I’m always reminded that I did my PhD in AI in the late 1990s, finishing in 2001. The situation was totally different then. We were still thinking about embodiment and embeddedness, especially here at Sussex, and some of the more in-principle limitations. But the practical capabilities of AI back then were just — there was nothing really to write home about. That’s changed so much. That’s why conversations like this now have real practical importance in the world.

AI is super impressive. I don’t see it as a single trajectory, though. I think there’s a meta-narrative we often fall into, which is that intelligence is along a single dimension — plants at the bottom, then insects, then other animals, then humans in a kind of scala naturae, the great chain of being — and then there’s angels and gods, and AI is travelling along this curve and at some point it’s going to reach human-level intelligence and then shoot past to artificial super-intelligence. I think this is a very constraining way to think of it.

It’s already the case, and has been for a long time, that AI has been better than humans at many things. But it’s always been very narrow. What we’ve seen through the foundation model revolution is the first kind of semi-general AIs — language models are good at many things, not good at everything, but good at many things rather than just one. But I still think they’re exploring a different region in the space of possible minds. They may soon be better than humans at many things, but they’ll still be different from us.

I think it’s important to recognise that, because we get into all kinds of trouble if — to use a beautiful metaphor from Shannon Vallor’s book about the AI mirror — we think of AI systems as just alternative instantiations of human minds that are either a little bit weaker or much stronger. Then we misunderstand both the systems and ourselves, and miss opportunities for how we can develop AI technologies so that they best complement our own cognitive capacities.

Dan Williams: Let’s go back to the consciousness issue. As you said, one reason you might think AI systems are or could be conscious is because of these cognitive biases. Another reason is you might hold a sophisticated philosophical view called computational functionalism. Can you say a little about how you understand computational functionalism and why it might commit you to the view that conscious AI is at least possible in principle?

Anil Seth: Yeah. So my understanding of computational functionalism is that it’s really an assumption you need in order to get the idea of conscious AI off the ground. It’s the idea that consciousness is fundamentally a matter of computation — and this computation is the kind that can be independent of the particular material implementing it. To put it another way: if you implement the right computations, you get consciousness. That’s sufficient.

That means if you can implement those computations in silicon, that’s enough. You could implement them in some other material — that would also be enough. It’s the computation that matters. The material underlying it is only important insofar as it’s able to implement those computations. And silicon is very good at implementing a certain class of computations — what we call Turing computations. So that makes it a good candidate for consciousness if computational functionalism is true. And that’s what I think is a big “if.” It seems a very natural assumption. But first let me ask you — does that resonate with your understanding of computational functionalism?

Henry Shevlin: I completely agree with that characterisation. Computational functionalism says mental states are individuated by their computational role. The only thing I’d push back on is that computational functionalism is one road to concluding that AI can be conscious, but there are other types of functionalism out there. My response to your BBS paper emphasises this.

Psychofunctionalism — apologies to listeners, the terminology does get messy — says we should individuate mental states not in terms of computational processes necessarily, but whatever functional roles those mental states play in our best scientific psychology. Ned Block is a big fan of this view. The view I’m partial to is analytic functionalism, which is the functionalist take on behaviourism: mental states should be individuated by everyday folk psychology. A belief is something we all sort of know what it is because we can characterise people as having them, forming them, losing them. Once you formalise this tacit knowledge, that gets you to a theory of what beliefs are.

Those views could overlap with computational functionalism, but it’s not necessary to endorse it to think AI is conscious. If you’re an analytic functionalist, you might think that if AI adheres sufficiently closely to the platitudes of everyday folk psychology — they believe like us, they form goals, they have hopes and aspirations — then of course they can be conscious, even if you think brains are not computers, even if what brains do is not a computational process and what AI systems do is. Because both fit the same functional-behavioural profile, they might both count as conscious.

Anil Seth: That’s quite a wrinkle — I’d say a massive fold. I completely agree that computational functionalism is a specific flavour of a broader set of functionalist views. Part of the problem has been that people assume all these views are equivalent, and they really aren’t.

Functionalism, as I understand the original version, just says that mental states are the way they are because of the functional organisation of the system. But that can include many things — the internal causal structure, many things not captured by an algorithm. An algorithm is in the end determined by the input-output mapping between a set of symbols. Functionalism in general can mean many other things. You could be a signed-up, subscription-paying functionalist and still disagree with computational functionalism, which is a much more specific claim about everything that matters about the brain being a matter of computation.

I’d also worry a bit about your view, Henry, which seems a little behaviourist. If you’re saying that behaving the same way and having the same kinds of beliefs are sufficient conditions — well, computational functionalism at least has the merit of specifically stating conditions for sufficiency. If you’re saying the same about folk-psychological criteria, I think you’re open to all the problems of the psychological biases we discussed. It’s a position that’s going to be much more open to false positives, because there are so many ways of things looking as if they have the kinds of beliefs and goals that go along with consciousness in us, but which need not go along with consciousness in general.

But back to the point: computational functionalism is this specific claim, grounded on the idea that the computation is what matters. And it’s also grounded on the idea that even in biological brains, it’s the computation that matters — and if you can abstract that computational description and implement it in something else, you get everything that goes along with the real biological brain.

Dan Williams: So roughly speaking, functionalism is the view that what matters for consciousness is not what a system is made of, but what it can do. And computational functionalism is the view that what matters in terms of what the system is doing is something like processing information.

Anil, your arguments have two aspects. Some are critical of computational functionalism — the negative part — and then you’ve got an alternative way of viewing consciousness and its connection to the brain. Let’s start with those criticisms. What do you think are the main weaknesses of computational functionalism?

Anil Seth: I think there are a number of weaknesses, all grounded on the intuition that we’ve taken what’s a useful metaphor for the brain — the brain is a kind of carbon-based computer — and we’ve reified it. We’ve taken a powerful metaphor and treated it literally.

The idea that the brain literally is a computer raises the question of what we mean by a computer, by computation. Let’s think of computation in the most standard way: as Turing defined it in the form of a universal Turing machine. In this definition, computation is a mapping between a set of symbols through a series of steps — that’s an algorithm. And this mapping involves a sharp separation between the algorithm and what implements it, between software and hardware. That sharp separation both influences how we build real computers — we can run the same software on different computers — and underwrites the assumption that computation is the thing that matters, because it allows you to strip out the computation cleanly from the implementation.

If you look at the brain, it has a superficial appeal: we think of the mind as software and the brain as hardware. But the closer you look, the more you realise you can’t induce anything like this sharp separation — not of software and hardware, but of mindware and wetware. In a brain, you can’t separate what it is from what it does with the same sharpness that, by design, you can in a digital Turing computer.

But Turing computation remains appealing. Roll back almost ninety years to Turing, but also to McCulloch and Pitts: they showed that if you think of the brain as very simple abstract neurons connected to each other, each just summing up incoming activity and deciding whether to be active or not — very simple abstractions of the biological complexity of real neurons — you basically get everything Turing computation has to offer. You can build networks of these that are Turing-complete; they can implement any algorithm.

So you get this beautiful marriage of mathematical convenience. You can strip away everything about the brain apart from the fact that it consists of simple neuronal elements connected together, and yet you get everything Turing computation can give you. So maybe that’s the only thing that matters about brains. And of course, that abstraction is in practice very powerful — the neural networks trained for foundation models are direct descendants of these McCulloch-Pitts networks.

But this marriage starts to get stressed, because Turing computation, while powerful, is not everything. Strictly speaking, anything that is continuous or stochastic is not within the realm of algorithms. Algorithms also don’t care about continuous time — there could be a microsecond or a million years between two steps; it’s the same computation. Real brains are not like that. We’re in time just as much as we’re embodied. You can’t escape real physical time and continue to be a functioning biological brain. The phenomenology of consciousness is also in time — time is plausibly an intrinsic and inescapable dimension of our phenomenology.

So there are things brains do which are not algorithmic and might plausibly matter for consciousness. And when you look at brains, you can’t separate what they are from what they do in any clean way. I think that really undermines the idea that the algorithmic level is the only level that matters.

To roll back to where we started: the idea that the brain literally is a computer is a metaphor. Like all metaphors, there’s a bit of truth to it. But not everything the brain does is necessarily algorithmic. And that opens the question: if we can’t assume everything the brain does is computational, that puts a lot of pressure on computational functionalism, which is based on the idea that consciousness is sufficiently describable by a computation.

Henry Shevlin: I agree with a lot of what you’ve said about the importance of fine details of realisation in brains. Peter Godfrey-Smith has also advanced this point, talking about the role of intracellular, intra-neuronal activity. Rosa Cao has had some great papers on this recently too.

But here’s a provocative analogy. Imagine we were trying to understand what art was, and all we had was paintings. We might say: clearly an essential part of being an artwork is pigment, because not only is pigment present in every example of art we’ve got, it’s essential to how it is artistic — pigment defines the formal properties of every piece of artwork we’ve ever seen. But of course, there are lots of types of art that don’t involve pigments.

In the same way, yes, all these fine details of wetware might be essential to the type of consciousness we see in humans and other animals, whilst not exhausting the space of possible conscious minds that might be very different from us.

Anil Seth: I think that’s fine. All I’ve said so far is that there’s the open question of whether things besides computation might matter, but then one has to give an account of what they are and why. If I wanted to make the case that some aspect of biology is absolutely necessary for consciousness, I have to do that separately.

These things are somewhat independent. Computational functionalism could be wrong, but biology could still be not necessary — there could be other ways of making art. If I’ve got a strong case that some aspect of biology is necessary for consciousness, then computational functionalism cannot be true. But the reverse is not the case.

Dan Williams: Maybe one question before we move on. I was a little confused reading your papers about which of the following two positions you’re defending. One position says: even if we could build computers that replicated all the functionality of a human being, it nevertheless wouldn’t be conscious. The other says: we just couldn’t build computers that replicate all of the functionality of a human being, because to do what human beings do, you need the kinds of materials and structures found within the brain. Those feel like two different positions. Someone could be a computational functionalist as a purely metaphysical doctrine, saying: if you could build a computer that does everything humans do, it would be conscious — it just so happens we can’t do that. Are you denying that metaphysical thesis, or making a different claim?

Anil Seth: There’s a lot in there. I am very suspicious of that metaphysical claim. Let me put it in a scenario that might help clarify.

Some people might say that if aspects of biology really matter, and we built a digital computer simulation including those details, would that be enough? We can do this ad infinitum — build a maximally detailed whole-brain emulation that digitally simulates all the mitochondria, even microtubules. Simulate everything. Would that be enough?

The metaphysical computational functionalist might say yes — somewhere in there, the right computations have to be happening. But I don’t think so, because it still relies on the claim that consciousness is constitutively computational. Making a simulation more detailed doesn’t make it any more real unless the phenomenon you’re simulating is a computation.

We make a simulation of a weather system; making it more detailed doesn’t make it any more likely to be wet or windy. Most things we simulate, we’re not confused about the fact that the simulation doesn’t instantiate the thing we’re simulating. If it is to move the needle on consciousness, that depends on the claim that consciousness is constitutively computational.

The irony is that if you think simulating the details is necessary — if you think you have to simulate the mitochondria — that actually makes it less likely that consciousness is constitutively computational. Because if consciousness is constitutively computational, those kinds of details should not matter.

A slight sidebar: I think this is ironically amusing because there are people investing their hopes, dreams, and venture capital into whole-brain emulation in order to upload their minds to the cloud and live forever. I think that’s very wrong-headed. If you think the details matter, then it’s unlikely consciousness is a priori a matter of computation alone.

So to your question: I’m very suspicious of that metaphysical claim. The burden of proof is on the computational functionalist to say why computation is going to be sufficient, given all the differences between computers and brains. I start from a physicalist perspective — consciousness is a property of this embodied, embedded, and timed bunch of stuff inside our heads. If you build something sufficiently similar, it will be conscious. The question is: how similar does it have to be? Does it have to be embodied? Made of neurons? Made of carbon? Alive? These are still open questions.

Henry Shevlin: Just to chime in — this point about simulated weather systems not getting anyone wet is obviously John Searle’s point originally. I think it’s better understood as a restatement of the disagreement rather than a dunk on functionalism. If consciousness is computational, then it is absolutely substrate-invariant. There are other things that are substrate-invariant: online poker is poker, online chess is chess, money is money whether it’s coins, banknotes, or on a balance sheet. So if consciousness is not computational, then a simulation won’t be conscious. But if it is computational, the simulation point has no bite.

Anil Seth: I don’t disagree. But the key point is: you can’t use the simulation argument to argue for the fact that consciousness is computational. If consciousness is computational, certain things follow about what happens in a simulation. But the fact you can simulate something doesn’t tell you anything about consciousness being computational.

I reread Nick Bostrom’s simulation argument paper while writing the BBS paper. He carefully interrogates his assumptions — that we don’t wipe ourselves out, that at least one person is interested in building ancestor simulations. But he also says: we have to assume consciousness is a matter of computation for this whole thing to get off the ground. And then he says, “Don’t worry, philosophers generally think that’s fine.”

Hold on a minute — that is the most contentious assumption by far of everything in the paper, and he gives it no critical examination. The fact that computational functionalism is at the very least contentious is, for me, very good evidence against the simulation hypothesis.

Dan Williams: I really want to get to your positive account, but one follow-up on your criticisms. One of your strongest arguments is that when you look at the brain, you don’t find anything like the hardware-software distinction central to digital computation as we understand it post-Turing. I think that’s true and important. But isn’t it possible that someone could say: that’s an interesting feature of how computation works in biological systems — people call it “mortal computation,” the term from Geoffrey Hinton — maybe having to do with energetic efficiency? But it doesn’t follow that you couldn’t replicate those computational abilities in digital computers. It could just be a contingent feature of our architecture.

Anil Seth: The first part is right, but the second part doesn’t follow. You can’t separate what brains are from what they do; there’s no sharp distinction between mindware and wetware. Rosa Cao has written about this, and there’s the notion of mortal computation from Hinton. Others have talked about biological computation, emphasising these features — you can call it generative entrenchment. I like the term “scale integration”: in biological systems, the microscales are deeply integrated into higher levels of description in a way that you can’t separate out. The macro and the micro are causally entangled with each other. This is very characteristic of evolved biological systems — there’s no design imperative from evolution to have a sharp separation of scales. And that has benefits: you get energy efficiency, and you may get explanatory bridges towards aspects of consciousness too, like its unity.

This is, for me, a very exciting avenue: if we stop thinking of the brain as just a network of McCulloch-Pitts neurons implementing some Turing algorithm, and start looking at what it actually is — what the functional dynamical properties of scale-integrated systems really are — I think we’ll learn a lot.

But the second part — that biological computation could be done in a digital computer — I don’t think follows, and this is why I resist calling these things varieties of “computation.” Whenever you use that word, it’s easy to slip into the idea that they’re portable between substrates. The biological computation my brain does in virtue of being scale-integrated could be simulated by a digital computer. But the simulation is not an instantiation unless what you’re simulating is constitutively that kind of computation. And biological scale-integrated computation is not digital Turing computation.

The more general point: the further you move away from a Turing definition of computation, the less substrate independence you have. Analog computers, for instance, implement features that are probably essential — like grounding in time with continuous dynamics — but they do not have the same substrate flexibility as digital computers. We love digital computers because they have that flexibility. But when it comes to understanding what brains do, whether in intelligence or consciousness, we can’t throw all these things away.

Henry Shevlin: A quick side note: the Open Claude instances, the more agentic Claude bots, have something called a “heartbeat” — a regular interval at which they can take actions. So we’re starting to see at least simulation of some temporal dynamics in large language models. Obviously radically different from the kind you’re concerned with, but interesting.

Anil Seth: I don’t buy that. That’s a simulated heartbeat. You could slow the clock rate down. You can give these things a sense of time, but it’s not physical time. Imagine you slow all the Anthropic servers way down — all the agents slow down, but the computation is still the same. We are embedded in physical time in a way that even agents with simulated heartbeats are not.

Dan Williams: I’ll set you up for developing your positive account with a question: well, isn’t computational functionalism the only game in town? Doesn’t it just win by default?

Anil Seth: No. That’s part of the issue — one of the responses is often, “What else could it be?” There’s a phrase, “information processing,” that I find increasingly revealing. It’s so common to describe the brain in terms of information processing that we don’t even realise we’re saying it, as if there’s no other game in town. What do we mean when we say a brain is processing information? It’s really not clear to me. The most rigorous formal definition is Shannon’s, which is purely descriptive — it doesn’t tell you whether a system is processing information.

But alternatives have been around for a long time. When I was doing my PhD at Sussex, there was the dynamical systems perspective, the whole enactive embodied approach to cognition — continuous dynamics, attractors, phase spaces. These describe complex systems doing things in ways which are not computational, not algorithmic. Brains oscillate — this is one of the most central phenomena of neurophysiology, as Earl Miller talks about a lot. And it would be crazy if evolution hadn’t taken advantage of this natural physical property. The right framework for understanding oscillatory systems is not an algorithm, because algorithms are abstracted out of time.

So there are many other games in town. A lot of these are perfectly compatible with functionalism, but now it’s a functionalism much more tied to the material basis — only some substrates can implement the right kinds of functions, and biological material may be necessary for the right kind of intrinsic dynamical potential.

I think biological naturalism is still basically a functionalist position. I’m wary of saying something considered vitalistic — there’s no magic, non-explicable, intrinsic quality about life associated with consciousness. Living systems can be distinguished from non-living systems in terms of functional description. Features like metabolism and autopoiesis are still amenable to functional descriptions, but now the functions are closely tied to particular kinds of materials, particular biochemistries. Metabolism is a function, but it’s a function inseparable from some material process. Maybe it doesn’t have to be carbon — maybe there are other ways of having metabolism. But you can always say that intrinsic properties at one level can be decomposed into functional relations at a lower level.

So I’m comfortable with functionalism broadly, but the question is: how far down do you have to go? And to Henry’s point: how do we make sure we’re not focusing on things that are contingently the case in biological consciousness only?

Many of the comments to my BBS paper said I haven’t made a rigorously indefensible case for biological naturalism, and I totally concede that. I don’t think there is one yet.

Henry Shevlin: Can I give you an opportunity to say more about autopoiesis specifically? I’ve yet to hear a really convincing case for how it helps explain what consciousness is. Here’s a dark framing. The standard Maturana and Varela notion of autopoiesis is a system continually replacing, maintaining, and repairing its own components.

A few years ago, I read about a horrific case: Hisashi Ouchi, a Japanese nuclear researcher who received the largest dose of radiation ever recorded. Every chromosome in his body was destroyed, no new cell production, no RNA transcription — his body couldn’t produce new proteins. Every cell was effectively dead; autopoietic processes had basically stopped. He was kept alive through amazing medical interventions — you could call it allopoiesis — for eighty-three days. And he was conscious and in a lot of pain throughout.

So here’s a case of someone in whom autopoietic processes had basically stopped, and yet he was still consciously experiencing severe pain. I’d love to hear more about why you think autopoiesis is important for consciousness.

Anil Seth: That is darkly, weirdly fascinating. Setting aside the horror of it — it would be very interesting to consider: has autopoiesis really stopped entirely, or is it winding down? I can imagine all sorts of problems with that dose of radiation, but it’s also not true that every cellular process stopped at the moment he was still alive for eighty-three days. It might be a gradual winding down.

If there were a case where you could show that all autopoietic processes had definitively stopped and yet consciousness was continuing, that would put pressure on the claim that autopoiesis is necessary in the moment for consciousness. It might still be diachronically necessary — systems have to have gotten those processes rolling to begin with.

The reason I usually mention autopoiesis and metabolism as candidate features of life is partly because they maximise the difference between living systems and silicon-based computers. They’re obvious examples of things closely tied to life, things that silicon devices clearly cannot have. It’s partly to emphasise how different these things are and why it’s very reductive to think of us as meat-based Turing machines.

There’s another reason to think about autopoiesis, and it’s the connection between autopoiesis, the free energy principle, and predictive processing as a way of understanding the contents of consciousness. There’s a line that can be drawn between these poles — what Carl Friston and Andy Clark and Jacob Hohwy have called the high road and the low road, but they meet in the middle.

The basic idea: start with the brain engaged in approximate Bayesian inference about the causes of sensory signals — very much a Bayesian brain perspective, Helmholtz’s “perception is inference.” Of course, Bayesian inference can be implemented algorithmically, but that doesn’t mean that’s how the brain does it. The free energy principle shows a way of doing it which follows continuous gradients — not necessarily an algorithm.

So our perceptual experiences of the self and the world are brain-based best guesses about the causes of sensory inputs. This doesn’t explain why consciousness happens at all, but gives us a handle on why experiences are the way they are. This applies to the self too: our experiences of selfhood are underpinned by brain-based best guesses about the state of the body — especially the interior of the body, through what I’ve been calling interoceptive inference. These processes are more to do with control and regulation. The brain, when perceiving the interior of the body, doesn’t care where the heart is or what shape it is — it cares how it’s doing at the business of staying alive.

This explains why emotional experiences are characterised more by valence — things going well or badly — rather than shape and location and speed. And prediction allows control: once you have a generative model, you can have priors as set points and implement predictive regulation to keep physiological variables where they need to be.

So far so good. We’ve gone from experiences of the world, to the self, to the interior of the body, from finding where things are to controlling things. And then comes the part that’s still difficult for me: that imperative for control goes all the way down. It doesn’t bottom out — it goes right down into individual cells maintaining their persistence and integrity over time. There’s no clear division where the stuff ceases to matter. And so you get right down to autopoiesis.

That’s where the free energy principle comes in. Living systems maintain themselves in non-equilibrium steady states — they maintain themselves out of equilibrium with their environment. To be in thermodynamic equilibrium with your environment is to be dead. By maintaining themselves in this statistically surprising state of being, they’re minimising thermodynamic free energy. And that becomes equivalent to prediction error in the predictive processing framework.

That’s the rough line. I’ll be very frank: there are bits along the way that can be picked at. One is the move from a thermodynamic interpretation of free energy to the variational, informational free energy interpreted as prediction error. There are results in physics linking thermodynamic and information theory, but do they do the job? Not so sure.

But it’s a reason to think about how you go from metabolism and autopoiesis all the way up to this broader frame for how brains work. There’s a phenomenological aspect too, which is speculative: if you try to think about what the minimal phenomenal experience might be, devoid of all distinguishable content — some meditators talk about pure awareness without anything going on at all — I’m a bit sceptical of that idea. I think it’s equally plausible that at the heart of every conscious experience is the fundamental experience of being alive. That is the aspect of consciousness that, for biological systems, is always there. Everything else is painted on top of that.

Peter Godfrey-Smith put it nicely in Metazoa: the more you think about what life is — these billions of biochemical reactions going on within every cell every second, electromagnetic fields giving integrated readouts — it’s much easier to think that that’s the kind of physical system which might entail a basic phenomenal state, compared to the abstractions of information processing. I think he’s on the right track.

The way to begin is to look at what are the functional and dynamical attributes of living systems at all scales and across scales, compared to other kinds of systems. Biochemistry is a big missing link — we tend to forget about it. Nick Lane at UCL is doing amazing work looking at mitochondria and anaesthetics and the deep biochemistry of what happens within cells — not only how anaesthetics work, but why the electric fields generated within mitochondria might join together to give a global integrative signal about the physiological state of an organism. Stories like this are where I see much more potential for building solid explanatory foundations for a biological basis of consciousness.

Henry Shevlin: A plus one for Nick Lane — huge fan. We should get him on the show.

Dan Williams: You’ve described a rich and fascinating alternative picture. One worry about the free energy principle approach, though: it seems too general. As people like Friston understand it, it applies at the very least to all living things, and maybe even more broadly. Most people want to say not all living things are conscious. And even in conscious organisms, many of these processes — ordinary facets of digestion, for instance — presumably don’t have anything to do with consciousness. These things are presumably still happening under general anaesthesia, and yet you don’t have consciousness. What we want from a theory of consciousness is some explanation of why some things are conscious and others aren’t, why certain states within conscious organisms are conscious and others aren’t. If you take this very broad framework, you’re not going to get that.

Anil Seth: You’re absolutely right. It’s why I resist saying the ideas I’m sketching constitute a theory of consciousness — they don’t, as they stand, do the job a good theory should do. A good theory should give an account of the necessary conditions, the sufficient conditions, and the distinction between conscious and unconscious states and creatures.

Biological naturalism, as I understand it — distinct from biopsychism — is a claim that properties of living systems are necessary but not necessarily sufficient for consciousness. Biopsychism is the claim that everything alive is conscious. I think that’s very strong; I wouldn’t want to defend it.

So what makes the difference? I think this takes us back to functions. We have to think about what the functions of consciousness are for us and for creatures where we can reasonably assume it’s there. That can move us from necessity towards sufficiency.

For me, every conscious experience in human beings seems to integrate a lot of sensory and perceptual information in a single, unimodal format centred on the body and our opportunities for action, strongly inflected by valence and with affordances relevant to our survival prospects, with particular temporal properties. It may be that when those functional pressures exist, they’re enough to make otherwise unconscious processes of autopoiesis and metabolism become a conscious experience. I don’t know — it’s partly an empirical question. For those functions to entail a conscious experience, you may need the fire of life underneath it all. I think that’s the idea.

Henry Shevlin: The question of sufficient conditions for consciousness in non-human animals is obviously very big for the ethical side. Whereas for AI, the necessary conditions are more relevant — if we can rule out that any of these systems are conscious, that makes the ethical situation a lot clearer. Since animals obviously satisfy the necessary conditions you’ve sketched, the question becomes which of them qualify.

A quick thought and then a question. I’m not sure whether your view is scientifically falsifiable. As you know, I’m very much a sceptic about the prospects of consciousness science as a falsifiable research programme. But maybe even setting aside strict falsifiability — what kinds of evidence would you be looking for over the next ten years that might push you in one direction or another?

Anil Seth: You can’t falsify a metaphysical position. Is biological naturalism a metaphysical position? It depends how much you flesh it out. I tend to be more Lakatosian in my view — I want things to be productive, not degenerate. Does unfolding the biological naturalist position lead to more explanatory insight? Does it lead to testable predictions and falsifiable hypotheses over time? If it does, that adds credence to the position, but it doesn’t establish it.

The position itself is not falsifiable as things currently stand, because we don’t have an independent, objective way of saying whether something is conscious. We always build prior assumptions in. Tim Bayne, Liad Mudrik, and I and others wrote a “test for consciousness” paper thinking of consciousness as a natural kind, but we’re always generalising from where we know — humans — outwards, trying to walk the line between taking contingent facts about human consciousness as general and expanding too liberally.

Evidence that would move the needle for me: to what extent can we demonstrate that properties of biological brains are substrate-independent? That’s a feasible research programme. We know some things the brain does are substrate-independent — that’s the whole McCulloch-Pitts story. But what about other things? What depends on the materiality of the brain? And what might be the functional roles of those things for cognition, behaviour, and consciousness?

Henry Shevlin: On the AI side, are there any predictions you’d feel comfortable about, or any evidence that might make you say, “This is evidence against biological naturalism”?

Anil Seth: The kind of evidence that would not convince me is linguistic evidence of AI agents talking to each other about consciousness. I can’t help being moved by it at one level — they’re very hard to resist, even if you believe they’re not conscious. It’s unsettling to hear these things talk about their own potential consciousness. But that’s not the right kind of evidence.

The more you can show that things closely tied to consciousness in brains are happening in AI, the more it would move the needle. For example, in a very influential paper, Patrick Butlin and Robert Long and others looked for signatures of theories of consciousness in AI models — does this model have something like a global workspace, or higher-order representations? They explicitly assume computational functionalism, looking just for the computational level of equivalence.

I think this is useful, but I’d try to drop that assumption and ask: how is a global workspace instantiated in brains at something deeper than just the algorithmic level? Do we have something like that in AI? This brings up neuromorphic computing — is the AI neuromorphic in a way that’s actually implementing, not just modelling, the mechanisms specified by theories of consciousness?

An issue is that most theories of consciousness don’t specify sufficient conditions. Global workspace theory is silent on what counts as sufficient for a global workspace. Higher-order thought theory doesn’t really tell you either. Ironically, the only theory that does is the most controversial one: integrated information theory. It explicitly tells you sufficient conditions — credit where it’s due, it puts its cards on the table.

Henry Shevlin: I’ve written a paper about exactly this — I call it the “specificity problem”: the difficulties of taking these theories off the shelf and applying them to non-human systems because they’re so underspecified. I actually call out IIT as one of the few non-offenders. But the downside is you end up with some very extreme predictions.

Anil Seth: Actually, me and Adam Barrett and others are writing a semi-critique of IIT. The expander grid thing is not as massively defeating as it seems, because in an expander grid, nothing is happening over time. You’d get something supposedly very conscious but of nothing — which is not a rich conscious state. But yes, it’s a non-offender on the specificity problem as you nicely put it.

Henry Shevlin: So to move on to the ethical side. Two big angles come up both in your paper and the responses to it. One is the danger of anthropomorphism and anthropocentrism — that we’ll see these things as conscious or develop highly dependent relationships with them. We’ve seen this at scale with social AI, AI psychosis, and so forth. The second is debates around artificial moral status — in your BBS paper, you talk about the danger of false positives and false negatives. And related to this is the call some people have raised, like Thomas Metzinger, for a moratorium on building conscious AI. A nice bouquet of issues for you to explore.

Anil Seth: I think there’s also a third element, which is how our perspectives on conscious AI make us think of ourselves — how it affects our picture of what a human being is. It’s more subtle but quite pernicious.

There’s an important distinction between ethical considerations that pertain to real artificial consciousness and those that pertain to illusions of conscious AI. Sometimes they overlap; sometimes they don’t.

If I’m wrong and LLMs are conscious, or if we build sufficiently neuromorphic AI that incorporates all the right features — I think this would be a bad idea. Building conscious AI would be a terrible thing. We would introduce into the world new forms of potential suffering that we might not even recognise. It’s not something to be done remotely lightly, and not because it seems cool or because we can play God. Thomas Metzinger talks about these consequences a lot. That’s one bucket.

The other bucket is illusions of conscious AI. This is clearly happening already. So many people already think AI is conscious, and none of the philosophical uncertainty matters — if people think it’s conscious, we get the consequences. These range from AI psychosis and psychological vulnerability — if a chatbot tells me to kill myself and I really feel it has empathy for me, I might be more likely to go ahead. That’s not great.

We also have this dilemma of brutalism. Either we treat these systems as if they are conscious and expend our moral resources on things that don’t deserve it, or we treat them as if they’re not, even though they seem conscious. And in arguments going back to Kant, this is brutalising for our minds — to treat things that seem conscious as if they are not. It’s psychologically bad for us. These illusions of conscious AI might be cognitively impenetrable. I think AI is not conscious, but even I feel sometimes that it is when I’m interacting with a language model — like certain visual illusions where even when you know two lines are the same length, they look different.

A good example where the ethical rubber hits the road is AI welfare. There are already calls for AI welfare, and firms like Anthropic are building constitutions for Claude and saying they take seriously the idea that their agents have their own interests in virtue of potentially being conscious. I think this is very dangerous. Calls for AI welfare give added momentum to illusions of conscious AI — people are more likely to interpret AI as conscious if big tech firms say they’re worried about the moral welfare of their language models.

And if we extend welfare rights to systems that in fact are not conscious, we’re really hampering our ability to regulate, control, and align them. The alignment problem is already almost impossibly hard. Why would we make it a million times worse by, for instance, legally restricting our ability to turn systems off if we need to?

And then there’s the image of ourselves. As Shannon Vallor writes about with the AI mirror — I think it’s really diminishing of the human condition. You mentioned the term “stochastic parrots.” It’s unfair on everything: unfair on AI, which is really impressive; unfair on parrots, who are fantastic; and unfair on us, because if we think a language model is a stochastic parrot and we also think that’s fundamentally what’s going on for us — that’s really reductive of what we are. That tendency to see our technologies in ourselves is a narrowing of the imagination of the human condition, and I worry about the consequences.

Henry Shevlin: I’ve got to flag one objection. You realise people make the same arguments about Darwinian evolution? That seeing us as just other animals is somehow diminishing to the human condition — that contextualising humans within the tree of life diminishes our dignity. I don’t agree with that argument, and I assume no one on this call does. But that strikes me as a worrying parallel for the kind of arguments you’re making.

I don’t think it diminishes human dignity to see us as continuous with the broader tree of life. And I don’t think it’s necessarily stripping human dignity to see ourselves as part of a broader space of possible minds, some biological, some very weird. We can preserve human dignity whilst making a more expansive vision of what intelligence and mind are.

Anil Seth: Maybe. It depends on your priors. I completely agree that seeing us as continuous with the rest of nature is actually very beautiful, empowering, enriching, and dignifying. And people often say: you’re very anti-AI consciousness, but people were anti-consciousness in animals too — look at the historical tragedy still unfolding through those false negatives.

My response is: I don’t think the situation is the same. There are reasons why we’ve been more likely to make false negatives in the case of non-human animals, and those same reasons explain why we’re more likely to be making false positives in the case of AI. Both have serious consequences.

Human exceptionalism is at the heart of both. It prevented us from recognising consciousness where it exists in non-human animals, and it’s encouraging us to attribute consciousness where it probably isn’t in large language models.

Having said that, the way I’d find your case convincing is this: just as there’s a wonder in seeing ourselves as continuous with many forms of life — we’re a little twig on this beautiful tree of nature — we can appreciate the singularity of the human mind and the human condition when we understand more about how different things could be, how different kinds of minds could be, whether they are conscious or not.

Dan Williams: I think that’s a great note to end on. I’m conscious of your time, Anil — otherwise we would just keep talking for hours. I really do hope you’ll come back in the future and we can pick up on one of these many threads. Thank you so much for giving up your time to come and talk with us today.

Anil Seth: It’s been an absolute delight. Thank you both for your time and for the opportunity. I think we did get into the weeds a bit, but I enjoyed that very much.

Henry Shevlin: Anil, it’s been an absolute delight personally, and I think we’re very lucky to have you on the show. This has been a fantastic conversation.

This is a guest post by David Pinsof, who writes the excellent ‘Everything is Bullshit’ Substack.

One of the great joys of intellectual life is finding someone to argue with in good faith. As someone who thinks most arguing is bullshit, it’s all too rare and precious to have a genuine exchange of ideas stripped of character attacks, strawmanning, and status jockeying. Thankfully, I think I’ve found such a good faith interlocutor in the ever-brilliant Dan Williams, who has written a moderately cynical yet optimistic essay in response to my soul-crushingly cynical and pessimistic essay, A Big Misunderstanding.

Dan’s post is called “We Are Confused, Maladapted Apes Who Need Enlightenment.” What Dan means by “enlightenment” is something like: “the culture and ideas of intellectuals.” And what he means by “confused and maladapted” is something like: “irrational, ignorant, self-deluded, and in dire need of the culture and ideas of intellectuals.”

My essay has a different message. I argue that we humans are pretty savvy and rational, shaped as we are by millions of years of natural selection, and that intellectuals often overstate the demand for their grand ideas, in large part by pretending we humans are confused and maladapted, so that they can cast themselves as humanity’s saviors.

So my response to Dan might be something like, “Yea, maybe humans are kind of confused and maladapted sometimes, but it’s also really insightful to see humans as savvy animals strategically pursuing their Darwinian goals.” And Dan might say something like, “Yea, it’s pretty insightful to see humans as savvy animals strategically pursuing their Darwinian goals, but it’s also really important to recognize that humans are confused and maladapted sometimes.” It’s basically a disagreement over where to put the italics.

But if it was all about italics, I wouldn’t be writing this. There are a few areas where Dan and I might truly disagree, which is very exciting to me. Maybe one of us or both of us will change our minds or come to see the world a bit differently. Maybe you, dear reader, will benefit from the back and forth. What a beautiful thing. Let’s go through what I see as the biggest potential sources of disagreement.

Subscribe now

Stone Age Minds in Modern Skulls

A big part of Dan’s post is about evolutionary mismatch. This is the idea that the human brain is primarily adapted to an ancestral environment of cave paintings and tribal warfare and saber-tooth tigers, which is very different from our modern environment of cellphones and skyscrapers and pornography. The lesson Dan draws from this is that if we’re so mismatched and maladapted, we could really use the help of intellectuals to tell us to put away our phones and read some economics. If we’re gorging on junk food that was scarce in ancestral environments, we might need a friendly reminder to eat healthy.

I take a different view. Mismatch is a thing, but it is increasingly being recognized by evolutionary psychologists to be overrated as an explanatory approach. I would know: I co-host Evolutionary Psychology (the Podcast) and talk to different evolutionary psychologists every week. The popular story about gorging on sugary or fatty foods that were scarce in ancestral environments has a bit of truth to it, but it’s too simple. A moment’s reflection will make you realize that we obviously have mechanisms for curbing our appetite when our stomachs are full, or when we’ve had too much bacon or fudge, or when we need to lie down because we’re in a food coma. If you don’t believe me, try eating nothing but Oreos for a couple days. You will feel like complete and utter dogshit. Your body will punish you; your mind will be driven to the brink of madness. Our evolved food psychology is more well-designed than the popular caveman story would suggest, in which the only thing stopping us from subsisting on Oreos is willpower. In reality, we have a variety of subtle cravings for specific nutrients that are sensitive to the vicissitudes of our diet and personal history and local ecology.

With regard to obesity, there’s a lot we don’t know, but part of the explanation may relate to the body’s tendency to store energy in the form of fat to ensure against the risk of future food shortages. Research by Daniel Nettle and colleagues suggests that obesity is more likely to occur when people experience skipped meals and food insecurity early in life, potentially explaining why poverty and obesity go together. It’s not that poor people lack willpower; it’s that their bodies are rationally stockpiling energy reserves when they’re getting cues that access to food is uncertain. If Nettle is right, then one could easily see why stress and obesity go together (thereby confounding the relationship between obesity and health), and why an obsession with dieting and fasting could tragically make matters worse.

But isn’t this just a different kind of mismatch story? I’m not sure: maybe stockpiling energy is still smart in the modern world, given that poor people really do face future food shortages, and given that civilization, the planet, and the international order are looking rather precarious right now. Maybe when catastrophic climate change or war with China or the AI apocalypse happens, fat people will inherit the earth. Regardless, what I like about Nettle’s hypothesis is that it avoids insulting the intelligence of both the evolutionary process and people living in poverty.

Then there is the story of ancestral, mobile, small-scale, egalitarian hunter gatherer tribes—another supposed example of mismatch to our swarming cities and towering wealth inequalities. Again, this story is too simple. Research by Manvir Singh and Luke Glowacki suggests that ancestral hunter gatherer societies were more variable in structure than is commonly assumed, with some being very large and very unequal. Singh and Glowacki have also gathered evidence from the ethnographic record to show that humans in forager groups often try to enforce the rules and social norms that personally benefit them, consistent with my cynicism about the intentions of intellectuals in the modern world. Finally, research by Duncan Sibbard-Hawkes and Chris von Rueden suggests that the “egalitarianism” found even among the most idyllic hunter gatherers has been greatly overstated, with many forms of brutal competition and hierarchy bubbling beneath the surface.

And as long as we’re on the subject of names you don’t know or care about, we should get into an important concept introduced by Clark Barrett: the difference between tokens and types. The idea is that we have cognitive adaptations to deal with particular types of things, like food, mates, groups, status, and zero-sum conflict. These adaptations help tailor our behavior to the particular tokens of those types we find in our current environment—the particular food items, groups, conflicts, mating opportunities, and status games we’re confronted with. The types of things we evolved to deal with are, for the most part, common to both modern and ancestral environments. We have groups now; we had groups then. We have status now; we had status then. We have politics now; we had politics then.

What’s more, many of these types are very broad, like “informative intentions” or “socially valued skills.” This enables unprecedented stuff to emerge, like sign language and constitutional lawyers. Then there are the various systems we call “reinforcement learning” or “predictive processing,” which provide us with additional tools to adapt our behavior to the novel tokens we’re confronted with in our lives, even tokens that are totally unprecedented in the history of life on earth. These learning systems can cleverly bundle together adaptations in new ways (like the bundling of object recognition and semantics that occurs with literacy), and they can turn amateur chess players into chessmasters who dream in pawns and rooks.

In other words, there are a lot of reasons to be skeptical of the idea that humans will be vexed, dumbfounded, flabbergasted, or ill-equipped to get their shit together in the modern world. Given the enormous range of social and physical environments our species currently inhabits, and likely inhabited ancestrally, it is a mistake to think there is one simple, caveman past that is tragically out of sync with the present moment. Our minds evolved in a bewildering variety of contexts, and part of the reason we have such huge brains is to reduce the bewilderment—to help us land on our feet in whatever urban or actual jungle we’re thrown into.

So if you’re tempted to call a human stupid for doing something that looks irrational, I think you should first ask yourself the question: “Am I grasping the entirety of that human’s situation, including all the relevant uncertainties and constraints?” If the answer is “yes,” then you should ask the follow-up question: “Is that human sufficiently incentivized to behave rationally in this context?” If the answer is “yes” again, then I would ask another follow-up question: “Am I correctly understanding that human’s motivations, including the motivations they may not want to admit to?” If you get another “yes” there, then sure, go ahead and call the human stupid. But please: don’t skip those first three questions.

Besides, even if it turns out that humans are woefully mismatched to the modern world—cavemen in suits, grunting their way through life—we have to ask ourselves another follow-up question: “Is there any reason to expect intellectuals to be more ‘matched’ than the masses?” The answer to this question is far from clear. After all, intellectuals have their own highbrow versions of junk food and misinformation.

Winners and Losers

Dan argues that one of the biggest sources of mismatch is in our zero-sum attitudes. Dan writes that “zero-sum thinking makes sense for hunter-gatherers. When you live at the subsistence level, one person’s dramatic gains likely mean someone else’s dramatic loss. Consequently, we struggle to comprehend how modern trade and innovation could make everyone better off, especially when gains are unevenly distributed or delayed.”

I think this is a good example of mismatch being overapplied. Dan is right that the modern world presents us with unique opportunities for wealth creation, but it also presents us with unique opportunities for cronyism, classism, cartelization, rent-seeking, censorship, surveillance, sectarianism, regressive redistribution, and regulatory capture. Status is zero-sum: when I rise, someone else falls. Political power is zero-sum: when the Republicans win, the Democrats lose. So once we correctly see wealth as an instrument of power-grabbing and status-seeking, it no longer seems like such a misunderstanding to view wealth in zero-sum terms. This is particularly true in a world where governments have interwoven themselves so much with capitalist wealth production that capitalism and politics can no longer be seen as separate entities. Perhaps our zero-sum mentality is exactly what we should expect to emerge in the sociopolitical system we currently inhabit, where political tribes cannot win at the same time, and where the winner gets to enforce its will on the loser by threat of imprisonment.

Of course, it would be better if our political system weren’t so high-stakes and zero-sum—with such a terrifying and enormous prize to fight over—but given that it is, we should not be surprised to see the masses rationally responding to it. Creating paranoid myths about conspiratorial outgroups is not stupid in this context: it is a good strategy for mobilizing one’s political coalition and gaining power—not to mention signaling one’s loyalties and jockeying for ingroup status. Just look at how well Trump’s preposterous bullshit worked out for him and his cronies: they’re some of the most powerful people in the world. Also, the left is hardly devoid of propaganda and has surely gained many political victories through fearmongering and Manicheanism historically. Yes, political elites can sometimes use propaganda to exploit the masses, but it’s important to remember that the masses often benefit from the propaganda too: political coalitions rise to power as a group, with both leaders and followers sharing in the victory.

The Arc of Progress

Dan talks about the positive trends in health, wealth, and safety that have occurred throughout history, citing the work of Steven Pinker and others. He views these salubrious trends as evidence that a kind of enlightenment has occurred—a march of progress led by the light of reason. I agree that positive trends have occurred throughout history, but I’m not so sure about the enlightenment bit.

I think it is a mistake to attribute these uplifting trends to any kind of conscious, overarching motivation for enlightenment. These trends must be explained in testable, mechanistic, incentive-based terms, like any other phenomenon in economics or social science. Ironically, viewing these trends as the product of conscious intent is the very same error of overattributed intentionality that Dan thinks the masses fall prey to. Insofar as intellectuals anthropomorphize “the enlightenment” as a brainy homunculus striving for a better world, it would be an example of intellectuals being just as cognitively mismatched as the masses—or, more plausibly, a case of them being biased toward self-aggrandizement.

So I don’t think the world became better because intellectuals got together and decided to help us all out of the goodness of their hearts. Instead, the world became better in the same way the world changes in any way at all: by people rationally responding to changing incentive structures. In this case, I would bet that the relevant incentives have more to do with expanding trade and global markets, which create wealth and break down tribal barriers, than with the good intentions of intellectuals, who often demonize markets.

Dan seems to agree on the importance of markets for explaining positive trends, citing the insights of Adam Smith and others, but then writes that “for this progress to be possible, societies require a critical mass of people to appreciate these insights.” I would disagree here: societies can get richer without anyone knowing or caring about Adam Smith. People do things that put cash in their pockets. No abstract theories from intellectuals are required for this to occur. Smith’s insights only emerged after the wealth-creating properties of markets were well underway, so he (or any other thinker) cannot take credit for producing them. I think the same is true of many other positive trends that intellectuals like to take credit for, including moral progress. Once you realize that markets pay people enormous sums of money to treat each other fairly and extend cooperation beyond tribal boundaries (as many different scholars have argued, including me), it begins to seem dubious that the primary cause of moral progress was a bunch of philosophizing.

Negative Nancies

Dan talks about our “deep-rooted negativity bias,” our “evolved (and, for hunter-gatherers, adaptive) tendency to attend disproportionately to threats and dangers.” He thinks that “the result is an information ecosystem systematically skewed towards catastrophe, conflict, and outrage.” The implication is that this negative skewing of reality is maladaptive in the modern world—something we should overcome with more enlightenment.

I disagree. If threatening, scary stuff is a kind of group mobilization fuel, then a good chunk of this catastrophism is politically rational. After all, it’s hard to mobilize a group by saying everything is peachy and getting better all the time. And once we realize that humans are not primarily dispassionate truth-seekers who care about accurately assessing intergenerational changes in health and income, but social primates who care about capturing each other’s attention, paying attention to what others are paying attention to, gaining and expressing sympathy for each other’s plights, signaling their competence and seriousness, being included in “important” conversations, demonizing their rivals, and saying things that are interesting and provocative, even our catastrophism has a certain kind of rationality to it. Positive trends are boring. Doomerism is exciting. Of course, all this doomerism takes a toll on our happiness. But as I’ve written about extensively (see here and here), we’re not pursuing happiness, so this shouldn’t count as evidence of human maladaptedness.

Mistakes Were Made (But Not By Me)

Dan lists a litany of bloody and catastrophic mistakes that have been made by humans throughout history, and I think this is where Dan is at his most persuasive. Humans have certainly done a lot of terrible shit. But while I acknowledge the tidal wave of stupidity that Dan is pointing to, it is important to remember that we are talking about the design of human nature, and how good we should expect that design to be—that is, whether the human mind is about as well-designed as the “hawk’s eye or the cheetah’s sprint,” as I put it in my post. When answering this question, we should not let ourselves get distracted by the specific failures of specific individuals, which are an inevitable part of life for any creature.

Predators often fail to catch their prey. Prey often fail to evade their predators. These failures cause death, which I’m told is a bad thing. But we shouldn’t conclude from these failures that predators and prey are dumb and irrational, or that they’re poorly designed for chasing and evading each other. “Haha, that gazelle just got eaten by a lion—what a dumbass!” In a world of fearsome competition and formidable constraints, deadly failures at the individual level and impeccable design at the species level are not mutually exclusive. Political revolutions often devour their children, but plenty of animals devour their children in the wild. The devouring does not necessarily make those animals, or their devoured children, maladapted.

Besides, even if we accept the Homo Stupidicus model that Dan is gesturing at, we have to ask ourselves the same question we asked previously: “Is there any reason to expect intellectuals to be less prone to these terrible blunders than the masses?” Given that many of the mistakes Dan cites were a result of intellectuals’ utopian visions, the answer is far from clear.

A Real Fixer Upper

Dan argues, contrary to my soul-crushing cynicism, that intellectuals often have a “genuine” motivation to fix the world. Rather than getting into a semantic debate about what it means to have a “genuine” motivation for something, I’d rather focus on what Dan and I seem to agree on: whenever people claim to be trying to fix the world, it is mostly because of deeper motives for esteem, prestige, admiration, etc. So if we want to understand this world-fixing business, we have to delve deeper into the prestige economy that gives rise to it. And once we delve deeper into that prestige economy, we will discover some serious grounds for pessimism. Because what gets a person prestige, and what fixes the world, are two very different things.

It is the appearance of world fixing to a prestige-granting audience—not objective world fixing in external reality—that intellectuals are striving for. And insofar as prestige-granting audiences do not actually know what fixes the world, or hold politically biased beliefs about what fixes the world, then intellectuals’ prestige striving will be uncorrelated with objective improvements in the world. You might even get a few cases where intellectuals get showered with virtue points for creating hell on earth. The disconnect between audience perceptions and objective reality is why I am more pessimistic than Dan about the world-fixing motivations of intellectuals. The lack of depth to these motivations is precisely what should make us skeptical that they will always lead to good outcomes, or that they are the main causes of moral and material progress throughout history.

Subscribe now

Enlightenment Now?

Aside from some differences in style, Dan and I probably agree on at least 90% of the substance, and I suspect he will be on board with most of what I’ve written here. That’s the beauty of good faith disagreement: it often reveals how little of it there is. And truth to be told, I’m just as enchanted by the ideals of the enlightenment as any other intellectual: it’s the animating force behind all my writing and researching and podcasting. So I really do get the emotional core of Dan’s essay. It’s what gets me out of bed in the morning.

But in spite of all the beauty and grandeur of the life of the mind, I cannot help but take a long hard look in the mirror and ask myself: “Is it all bullshit?” I think it’s important to ask ourselves this question, and if we’re going to ask it, we must be genuinely and uncomfortably open to the possibility that the answer is yes.

Does ‘wokeness’ exist at all? And if it did, why on earth would anyone get worked up over it? Over the past decade, many mainstream British liberals have reacted to the rise of ‘wokeness’, or what we prefer to call social justice ideology, by denying that anything of note is occurring. These kinds of denials have taken many forms. Among those we could affectionately term ‘centrist Dads’, there has been a rich seam of indignation that these sorts of questions are being raised at all. ‘Why do you care?’ ‘Why are you so obsessed with this?’ ‘Stop spending so much time on the internet!’ We have heard many responses along these lines.

On a more intellectual level, liberal objections have sought to characterize social justice ideology as just a particularly earnest and sincere form of liberalism. Surely, we hear, now is not the right time to look into this issue, given the very real threat of radical right populism? At worst, the desire to ask these questions at all is seen as alarming evidence of authoritarian tendencies. ‘It’s just idealistic kids’, we are told. Ignore it. It will all pass over in time. In academia, this has engendered a curious phenomenon: a near dearth of accounts which examine social justice ideology through an analytic lens.

In our new book, Beyond Woke and Anti-Woke: Explaining the Rise of Social Justice Ideology (Bristol University Press, 2026), we examine the seeming inability of liberals to describe their left flank in accurate terms (or even to admit that it exists at all). Over the past decade, liberalism has palpably lost ground to ‘woke’. Both are progressive ideologies but, in contrast to liberalism, social justice ideology emphasizes the overriding importance of identity and direct action. It extends the concept of harm far beyond previous limits, concerning itself particularly with the threats of emotional harm and harmful speech. These developments have brought social justice advocates into conflict with older liberal tenets: individualism, legalism, and freedom of speech and association.

This conspicuous gap in the scholarship is curious because, in academia, analytic approaches to ideology are common. To give a well-known example, there is an extensive literature on radical right populism which, over thousands of studies, examines the origins and trajectories of this ideology. Admittedly, many who are interested in the study of social justice ideology have not helped their cause. Since the mid-2010s, a self-described ‘heterodox’ movement has arisen that has sought to investigate the topic. This movement has produced some important works, yet has ultimately failed to grasp the social justice nettle. We have seen a great many polemical trade-press books and podcast episodes and many ‘free speech’ festivals at which speakers celebrate their ability to tolerate robust disagreement (but at which no one seems to disagree at all). Despite their initial energy and sense of purpose, many in this ‘heterodox’ space have ultimately abandoned all pretence of academic rigour. While surrendering to the temptations of audience capture might be good for Substack subscriptions, it makes it easier for liberals to dismiss these kinds of interventions. No smoke; no fire; nothing to see here.

Subscribe now

What, ultimately, explains the reluctance of liberals to acknowledge the existence of social justice ideology? We can think of some potential reasons: a certain hesitancy to expose the fractures in progressive movements, an unwillingness to be seen siding or affiliating with conservatives on certain issues, and (perhaps) fear of attack from radical activists. Sometimes, these kinds of motivations seem to be accompanied by the notion that progressive ideologies do not need to be explained. As classic theories of ideology contend, the ideologue regards their own worldview as an accurate depiction of reality and, therefore, without any need of further explanation. The overt similarities between liberalism and social justice ideology, and the obvious differences that separate them from conservative ideologies, encourage such thinking.

We find this state of affairs unfortunate, and it is what moved us to write our book. Just like radical right populism, we believe that social justice ideology should be studied with an analytic approach. Since the 2010s, social justice ideology has been the major newcomer in progressive ideological space and is notably different from liberalism. This development is fascinating and, rather than unevidenced polemics, the ideology deserves a serious programme of academic study, just like other ideologies.

In Beyond Woke and Anti-Woke, we explain the emergence of social justice ideology using statistical analysis of multiple surveys of UK and US public opinion, institutional theories of the political economy and morphological theories of ideology. Rather than having one cause, social justice ideology in fact reflects a wider demographic revolution. Mass higher education has transformed societies and, as women have entered public life, feminine-coded values of care and equality have become increasingly influential.

In particular, the crises of capitalism after the 2008 financial crash acted as catalysts for ideological change, giving social justice ideology mass appeal. Though our statistical analyses cast doubt on there being any direct relationship between adherence to social justice ideology and individual economic precarity (contrary to popular theories), we argue that economic crises helped discredit liberalism among younger groups. For corporations, the embrace of social justice ideology provided renewed legitimacy. By the 2020s, social justice ideology had become a major rival to liberalism and, notwithstanding the attacks of the second Trump administration, it remains a major force in progressive politics.

Of course, any such interpretations must be provisional. Beyond issues with replication, the lack of prior research on this topic makes conclusions unusually precarious. Will this change in the future? Though proponents of a spatial hypothesis expect that academic supply will inevitably meet any gaps that appear in the intellectual market, we are less optimistic. Academic fields are path dependent and, therefore, tend to follow their own logic. If they have been closed off to certain lines of inquiry in the past, there is a certain inevitability that they will continue to be so. Moreover, we know from survey data that progressives are numerically predominant in universities. Inevitably, this will create pressures from within fields and disciplines to maintain existing path dependencies.

Perhaps there is need for a cultural change in this area. Contrary to the fears of some, analysing progressive ideologies does not imply that one regards them as a problem to be solved. Instead, this sort of investigation should be entirely conventional in academia; one identifies a gap in the research and, using established theories and methods, arrives at findings. In the case of social justice ideology, its crucial influence on institutions and policymaking adds to the justification for such a research agenda.

This, we argue, would not only lead to a healthier academia; it would lead to a healthier liberalism.

In a characteristically insightful and entertaining essay, David Pinsof argues that intellectuals greatly overestimate how many of the world’s problems stem from popular misunderstandings. In reality, Pinsof argues, people are highly rational and well-informed about their interests. This is what we should expect on evolutionary grounds. “Show me an animal that has succeeded in surviving and reproducing in a hostile environment for millions of years, and I will show you a rational animal.” It is also supported by extensive evidence about the rationality and accuracy of human cognition.

In Pinsof’s worldview, even the dreaded cognitive “biases” that psychologists love to tell us about function as adaptive mechanisms that help us survive and thrive. Confirmation bias, for example, provides us with intellectual ammunition for persuasion and reputation management, while overconfidence and self-serving illusions help us win friends and influence people.

Why, then, do intellectuals so often chalk up the world’s problems to mass ignorance and irrationality? Partly, the narrative is simply self-serving. It is intellectuals, after all, who promise to liberate us from misunderstanding. They are our professional understanders.

But it’s also because they confuse our expressed motives with our real goals. Sure, Pinsof concedes, we look pretty stupid and misinformed relative to the high ideals and noble ambitions that we say we have. If we’re chasing objective truth, impartial justice, and effective altruism, we’re not doing a good job. But those goals are just elaborate fictions, self-serving public relations cooked up to make us look good. Our real goals, our hidden motives, are very different. We’re chasing the kinds of grubby rewards you would expect of apes forged in Darwinian competition: status, reputation, power, sex, and resources. And relative to those ambitions, we’re smart and sophisticated.

This analysis reframes many apparent examples of stupidity as strategies. For example, “tribalism” isn’t a cognitive error to be remedied by debiasing and education; it’s a winning strategy among groupish primates who care more about power and prestige than truth or justice. Ineffective altruism and slacktivism don’t result from miscalculating the most effective ways to help others; they help status-seeking activists buy noble reputations at a discount.

Unsurprisingly, this perspective leads Pinsof to a bleak conclusion. If most of the world’s problems result not from misunderstandings but from conflicting incentives, intellectual enlightenment cannot save us. And even if it could, nobody really cares about solving the world’s problems anyway:

Not every problem has a solution. Some things cannot be fixed. And once you come to the bracing realization that we have no deep desire to fix our broken world, you’ll realize that our problem is that we have no problem. What’s broken is that nothing is broken. The study of human nature is, all too often, the study of the hole we’re stuck in… In the end, the only misunderstanding is that there’s been a misunderstanding.

A Darwinian Defence of the Enlightenment

It’s a beautifully cynical, Pinsofian analysis—and one that, I think, gets a lot right.

Nevertheless, it is too optimistic about our baseline rationality. Yes, we are savvy and strategic primates pursuing goals we’d rather not admit, even to ourselves. But we’re also riddled with costly cognitive biases, maladapted to the modern world, and in need of enlightenment by intellectual knowledge that we often find deeply counterintuitive.

It is also too pessimistic. Some people really are motivated to fix our broken world, and in some cases, they make genuine progress. The motivation is never very deep or pure—no straight thing was ever made from the crooked timber of humanity—but it’s not merely a deceptive story, either.

There are many holes we will never escape from. There is an unavoidably tragic aspect to the human condition. But when scaffolded by the right incentives and error-correction mechanisms, we can draw on intellectual knowledge and cooperation to climb out of the worst pits we find ourselves in.

You can’t understand much of humanity’s significant progress over the past several centuries—in life expectancy, living standards, wealth, health, infant mortality, freedom, political governance, and so on—without embracing this fundamental optimism of the Enlightenment. Or so I will argue.

Evolutionary Expectations

Before getting into the details, it’s worth stepping back and scrutinising Pinsof’s assumptions about evolution and human rationality. He says,

“The default assumption of every intellectual should be that the human mind is about as well-designed as the hawk’s eye, the bat’s sonar, or the cheetah’s sprint.”

Our species complicates this default assumption in two ways.

A Uniquely Unique Animal

First, although all species are unique, it’s not just human chauvinism to think that we’re uniquely unique, a genuinely new kind of animal.

There is no single quality responsible for this—no magic bullet that set our ancestors on a novel evolutionary pathway. Instead, there is a set of interacting traits connected to our unique capacities for cognition (how we think and reason), cooperation (how we work together), and culture (how we share and accumulate information). Through such abilities, we have acquired unprecedented powers to design and redesign our environments, but we have also become vulnerable to novel risks and failure modes.

To take only one example, no other species is anywhere near as dependent on lifetime learning as we are, including extensive “social learning.” To achieve our goals, we rely on information acquired from others (parents, family, friends, allies, teachers, shamans, priests, Substackers, etc.), typically because they intentionally share it with us through language and other forms of communication.

Given this reliance, evolution has endowed us with highly sophisticated social learning mechanisms. In this sense, Pinsof is right that evolutionary theory correctly predicts rationality and adaptation. We’re skilled at extracting knowledge from others while minimising the risks of misinformation and deception. Even from a young age, we instinctively evaluate the plausibility of what we’re told, assess people’s reliability and honesty across different domains, and insist on persuasive arguments for surprising claims.

Nevertheless, such extensive social learning also creates novel vulnerabilities that won’t be illuminated by analogies to the hawk’s eye, bat’s sonar, or cheetah’s sprint.

Most obviously, it means that reflection on human evolution should never be used to discount the importance of ideas. We evolved to be a species dependent on good ideas—on the knowledge, wisdom, and understanding that we acquire from others. If such ideas are misleading or deceptive in ways we can’t anticipate or detect, even optimal learning mechanisms won’t prevent us from being misinformed in costly and sometimes catastrophic ways.

Before the Neolithic Revolution, this vulnerability wasn’t very pressing for most humans. The challenges hunter-gatherers faced were mostly local and small-scale: which plants are edible, which animals migrate, which group members are trustworthy, and so on. This meant they could often cross-check what they were told against direct experience.

Moreover, because our core intuitions evolved over hundreds of millennia in response to hunter-gatherer lifestyles, people’s instinctive bullshit detectors were broadly reliable in such domains. Whenever they encountered claims that seemed implausible or outlandish—that is, counterintuitive—they could usually safely dismiss them, or at least insist on practical demonstrations of their veracity.

Finally, because their social networks were mostly face-to-face, highly interdependent, and largely egalitarian, high-stakes deception was typically risky and counterproductive. When everyone knows everyone extremely well, and power is broadly distributed, it’s easier to discover and punish big lies. And when everyone depends on everyone else for the most basic necessities of survival, the social costs of getting caught lying can be astronomical.

Of course, hunter-gatherers believed plenty of preposterous falsehoods about matters beyond their experience—for example, about the broader cosmos, their ancient history, or the character of rival tribes. But such myths were generally costless and adaptive. When you lack the ability to influence the world beyond your immediate, day-to-day existence, you can believe whatever you want about it, which is exactly what they did.

The New World

Well, things have changed. The second reason humans complicate the link between evolution and rationality is that the modern world we must navigate is unimaginably more vast, complex, and unequal than hunter-gatherer environments. As John Dewey observed a century ago,

“The local face-to-face community has been invaded by forces so vast, so remote in initiation, so far-reaching in scope and so complexly indirect in operation that they are, from the standpoint of the members of local social units, unknown. . . . They act at a great distance in ways invisible to [them].”

Natural selection adapts organisms to their environments. When these environments change, such adaptations can become “mismatched.” This is why things aren’t going so well for polar bears.

In the human case, evolutionary mismatch is often invoked to explain relatively mundane things, such as why so many of us are obese. As the familiar story goes, sugar and fat were scarce in ancestral environments, so we evolved to crave them. In modern capitalist societies, they are abundant. So we gorge on cheesecake and pizza served to us by profit-seeking companies that place no value on our welfare. There is no savvy strategy behind such overeating. Most of us are simply heavier and unhealthier than we would like to be.

This basic lesson generalises to many other contexts, including those where our maladaptation is harder to observe than our fatness. The most important of these is modern politics.

Political Mismatch

The scale and complexity of the modern environment that bears on political debate are mind-boggling. Hundreds of millions of strangers are enmeshed in interacting economic, political, and institutional forces that bear no resemblance to the small-scale worlds we evolved in.

Although it’s important not to overstate the problem of mismatch here—popular talk of static “stone-age minds” obscures how we evolved to be highly adaptable and flexible—it’s equally important not to ignore the severity of the challenges.

First, the modern world radicalises our reliance on social learning. When forming beliefs about topics relevant to modern politics, we almost always lack the ability to cross-check what we’re told against our experience, either because it is too distant in space and time or because the topics concern abstract phenomena (GDP, inflation, demographic trends, economic growth, etc.) that no one can directly experience.

Second, the intuitions most people bring to understanding modern societies are systematically misleading.

We evolved to be highly skilled at forming alliances, reading intentions, tracking reputations, and playing local status games. In contrast, neither our evolutionary endowment nor first-hand experiences prepare us to understand large-scale systems characterised by emergent properties, distributed processes, and incentives. So we anthropomorphise institutions and frequently default to moralised, intention-based narratives that posit villains rather than incentives and structural constraints.

When pharmaceutical prices rise, we assume that greedy executives are to blame rather than laws, regulations, and insurance markets. When housing or renting becomes unaffordable, we blame the avarice of developers and landlords rather than building restrictions, permitting processes, and construction costs. Such tendencies served our ancestors well. In hunter-gatherer societies, it’s reasonable to trace significant events to identifiable agents with familiar goals, and to link good and bad social outcomes to good and bad intentions. Invisible-hand coordination and emergent order are, therefore, deeply counterintuitive.

Similarly, zero-sum thinking makes sense for hunter-gatherers. When you live at the subsistence level, one person’s dramatic gains likely mean someone else’s dramatic loss. Consequently, we struggle to comprehend how modern trade and innovation could make everyone better off, especially when gains are unevenly distributed or delayed. In fact, the very idea that something called “wealth” can be created is a profound theoretical discovery that conflicts with common sense. The more natural view, which modern economics education tries to shake people out of, is that there is a fixed set of goods to be distributed.

Third, the modern information environment through which people attempt to learn about this strange, new world and overcome their default ignorance and confusion is more of a hindrance than a help.

Most obviously, it is shaped by extremely well-funded propaganda campaigns by powerful strangers who profit from other people’s ignorance. Even if such propaganda is unsuccessful, as it often is, its presence can breed pervasive mistrust that prevents the uptake of trustworthy information, causing people to place greater weight on their personal—and highly unreliable—intuitions.

Even more importantly, the modern media environment in free societies is organised around intense competition for audience attention and engagement. When combined with our deep-rooted negativity bias—our evolved (and, for hunter-gatherers, adaptive) tendency to attend disproportionately to threats and dangers—the result is an information ecosystem systematically skewed towards catastrophe, conflict, and outrage.

The predictable consequence of this is that people develop mental pictures of reality far more negative than the objective facts warrant. They overestimate poverty, crime rates, and many other social pathologies and dangers, and believe most trends are going in the wrong direction. In affluent liberal democracies, people are not just largely oblivious to progress. Their minds invert reality, often treating the most peaceful and prosperous societies in human history as dystopian hellscapes.

The result of all this is pervasive ignorance and misperception. The facts and complexities of the modern world are substituted in people’s heads with cartoonish, catastrophising myths.

The Rational Irrationality Objection

If this analysis is correct, it suggests that mass ignorance and misperceptions are not figments of intellectuals’ self-serving imaginations. Evolution made us rational and well-adapted—but to a world that no longer exists. In the modern world, confusion and misunderstanding are the default.

Nevertheless, there is a popular line of reasoning that concedes the existence of mass ignorance but insists that it is “rational.” To introduce a bit of jargon, it acknowledges that most people are not “epistemically rational”—they are doing a terrible job forming accurate beliefs about reality—but it argues that such epistemic failures are “instrumentally rational”. In line with Pinsof’s perspective, it treats widespread error and delusion as an adaptive response to people’s practical circumstances.

One influential theory of this kind comes from the work of economists like Anthony Downs and Bryan Caplan. It points out that in large-scale modern democracies, an individual’s vote makes practically no difference to electoral outcomes. This means people have no incentive to become well informed. They have no skin in the game. On the other hand, endorsing political beliefs that are emotionally gratifying or that signal one’s tribal loyalties can be highly beneficial. So rational individuals opt for ignorance and (epistemic) irrationality.

This analysis could be strengthened by Pinsof’s “Alliance Theory” of political belief systems, which posits that people’s participation in politics is not rooted in a desire to form accurate beliefs. Instead, we’re tribal propagandists. Our beliefs are downstream of the alliances and rivalries we form, and the biased, hypocritical arguments we construct to make our allies look good and our rivals look bad.

Both perspectives are insightful, but they also go too far.

Sometimes Ignorance and Irrationality Are Just Ignorance and Irrationality

One problem for the “rational ignorance” perspective is the prediction that ignorance and misperceptions will evaporate when people have skin in the game. This is wrong.

The history of modernity is littered with examples of people making catastrophic decisions based on deranged, inaccurate worldviews in high-stakes contexts. The Nazis really believed in an elaborate Jewish conspiracy, which led them to undertake self-defeating decisions, such as diverting crucial wartime resources to mass genocide. As I will return to below, communists throughout the twentieth century genuinely believed in various myths about human nature and economics, which led to repeated catastrophes, many of which engulfed the revolutionaries who brought such regimes into existence.

For less severe examples, one need only look at the policy track records of populist politicians in modern democracies to see that people often make bad decisions based on ignorance and misperceptions, even when they have strong incentives to perceive reality accurately.

The idea that political cognition improves dramatically as stakes increase is not well supported by the historical record. And once you reflect on the vastness, complexity, and inaccessibility of the modern world, this shouldn’t be very surprising. When discovering the truth is extremely challenging, merely increasing people’s incentive to discover it won’t secure success.

Another problem with the “rational ignorance” perspective is the assumption that people know their individual vote has no impact on political outcomes and so “decide” to be ignorant and misinformed. As Jeffrey Friedman points out, this isn’t well-supported by evidence. Instead, people appear to be radically ignorant, not rationally ignorant. Because they don’t instinctively appreciate the sheer scale of the modern world, they dramatically overestimate the impact of their vote, and they treat political knowledge as much more accessible than it really is.

This analysis helps to explain many features of political psychology that sit uneasily with the “rational ignorance” perspective. The intensely negative, catastrophising worldviews that many people develop often just make them sad, distressed, and demotivated. They experience politics as aversive and anxiogenic. They sometimes damage close relationships with friends and family members. Much of this looks more like sincere participation than tribal signalling optimised for maximising emotional or social rewards.

None of this is to deny that people approach politics with a “tribal” mindset. There is considerable insight in Pinsof’s analysis that politics is rooted in alliances, rivalries, and self-serving (well, alliance-serving) “propaganda,” as well as in the popular idea that much political participation is performative, concerned more with tribal signalling than sober policy analysis.

However, the problem with such proposals is that the leaders and tribes we support and oppose are not independent of—in technical jargon, they’re not “exogenous” to—our political beliefs, so they cannot fully explain such beliefs. We choose leaders, allies, rivals, and enemies based on the pictures in our heads. If those pictures are systematically warped by misleading intuitions, mistrust, and negativity bias, the same will apply to our judgements about which leaders and allies promote our interests.

Put simply, someone with an accurate, evidence-based worldview will support very different political leaders and tribes than someone whose worldview is constructed from “common sense” intuitions interacting with their TikTok feed.

In general, political ignorance and misperceptions aren’t always or even commonly the product of savvy, evidence-based cost-benefit analysis or 4D Darwinian chess. They’re often downstream of the profound challenges of acquiring counterintuitive knowledge in a hostile and misleading information environment.

Subscribe now

The Role of Intellectuals

This suggests a more optimistic assessment of the value of “intellectuals” in the broad sense of that term (scientists, statisticians, academics, etc.), and of the kinds of knowledge they can provide, ranging from carefully collected data to rigorous scientific inquiry. To successfully navigate the modern world, we need to be enlightened by such knowledge. It won’t fall into our lap if we let our evolved psychologies run on autopilot. Our default condition is one of epistemic darkness.

This optimism is, or at least should be, uncontroversial when it comes to the knowledge associated with the natural and medical sciences. Hundreds of millions of people died throughout history from diseases we have now eradicated thanks to discoveries about vaccines and other miracles of modern medicine and public health. We couldn’t rely on Darwinian adaptations to secure such knowledge. We needed rigorous, institutionally supported inquiry through which we could learn truths that are often highly counterintuitive.

The real controversy concerns whether intellectual knowledge can correct costly ignorance in domains like politics and collective organisation.

Here, scepticism is understandable. It’s much more challenging to conduct rigorous science in these domains, and prominent ideas and theories often function more like intellectual fashions governed by the subjective, internal criteria upheld by the intelligentsia than like scientific hypotheses evaluated by objective measures of predictive success.

For this reason, the practical track record of these ideas has often been negative, and in some cases disastrous. Despite concerted and ongoing obfuscation of this fact by many left-wing intellectuals, the clearest example lies with Marx, who, alongside many later generations of communist intellectuals and activists inspired by his work, argued that self-interest and social competition were not essential features of human nature but contingent products of exploitative economic systems like capitalism, feudalism, and slavery. This and countless other foolish ideas, such as the notion that law and conventional morality under capitalism are mere “bourgeois prejudices,” played a major and undeniable role in many of the worst human catastrophes of the twentieth century.

These catastrophes can’t be understood as moral abominations that nevertheless advanced the strategic interests of those who spread them. Most of the true believers who fought for communist revolutions in countries like Russia, China, Korea, and Cambodia were quickly victimised by the systems they helped create. They weren’t just playing cynical adaptive games. They were catastrophically misinformed about reality in ways that got themselves and countless others killed.

Notice, however, that one should not conclude from such disasters that intellectual ideas don’t matter. They matter enormously. But wouldn’t it be strange if they could only have negative consequences?

The Achievements of Liberalism

In fact, one can find many examples throughout history of intellectual achievements concerning society and politics that have had extremely beneficial consequences.

For example, as Steven Pinker, Joseph Heath, Jonathan Rauch, and many others have documented, one cannot understand the emergence of modern liberalism and the unique social and political successes of liberal states without appreciating how complex, counterintuitive intellectual discoveries informed institution-building. From at least Hobbes onwards, a tradition of intellectual inquiry—including Locke, Hume, Montesquieu, Smith, Kant, and many other Enlightenment thinkers—drew attention to two major theoretical insights.

The first was that human societies are pervaded by what social scientists now call “collective action problems”: situations where individuals acting on their rational self-interest are led to engage in collectively self-defeating behaviour that leaves everyone worse off.

For example, Hobbes observed how, in the absence of enforceable laws and contracts, people who would benefit from mutual cooperation would be driven towards pre-emptive aggression, fearing exploitation or cheating by others. Insights with a similar structure were later used to explain the value of political regimes that uphold religious and political tolerance, enforce extensive systems of individual rights, protect free speech even for dangerous and heretical ideas, maintain open trade between nations, and more.

The second insight was that institutions can be constructed to channel self-interest and social competition away from predation and violence towards beneficial outcomes.

For example, Smith demonstrated how regulated market competition could transform the self-interest of bakers and brewers into the efficient production of goods for others. In the political domain, Montesquieu and Madison explored how political systems could be organised to make ambition counteract ambition. And in the domain of knowledge, many scientists and philosophers explored how formal societies and norms could be crafted to counteract individual biases and allocate “credit” only to those who made genuine discoveries. The core discovery across these diverse contexts was that specific systems of norms and institutions can convert human self-interest and ambition into innovation, investment, knowledge, and political accountability.

In both cases, these insights were genuine theoretical discoveries that sharply contradicted most people’s intuitions. Trusting in self-interest, social competition, and decentralised markets to coordinate economic activity; relinquishing power to people with radically different political or religious views; tolerating dangerous and offensive speech—these ideas don’t come naturally to human beings. They are insights that must be achieved.

Of course, the insights alone don’t change anything. Merely recognising the existence of a collective action problem doesn’t free one from it. And merely understanding that institutions can channel ambition into cooperation doesn’t create them. Nevertheless, precisely because these insights take humans as they are, not as we’d like them to be, and point to possibilities that leave everyone better off, not just some people, they can guide institutional design and tinkering, helping humanity gradually escape from the poverty, ignorance, and conflict that are our default state.

Notice, however, that for this progress to be possible, societies require a critical mass of people who appreciate these insights. Similarly, there must be enough people who have a grasp of the basic facts and trends demonstrating that such insights work—that, as a consequence of liberal institutions guided by intellectual insights, much of humanity has experienced objective progress along countless dimensions, including wealth, health, freedom, opportunity, governance, and much more.

The Desire to Fix the World

Reflecting on this history puts pressure on Pinsof’s pessimistic judgement that nobody really wants to fix our broken world.

Once again, there is more than one grain of truth here. Humans are unavoidably self-interested and competitive, and altruistic motivations are inevitably limited and accompanied by a large dose of selectivity and hypocrisy. This is what we should expect on evolutionary grounds, and it is confirmed by extensive historical evidence, not least the many examples where revolutionaries championing justice have quickly turned into despots after taking power.

Nevertheless, the historical record suggests that the deep human craving for esteem and honour can also be channelled into genuinely noble pursuits. As we created liberal societies that increased the scale of cooperation and the costs of predation, we also created conditions that made the pursuit of prestige—of admiration and deference—more profitable. As Will Storr documents in The Status Game, this channelled insatiable human ambition and social competition towards impressing others through demonstrations of competence and virtue, fuelling modern science, innovation, and social justice.

We reward those who try to fix the world, produce novel insights, and advance other people’s interests. At the same time, we are sensitive to the possibility that such motivations aren’t genuine—that people care only about the personal rewards, not the high ideals. Nichola Raihani calls this the “reputation tightrope”: to earn a noble reputation for performing good deeds, those deeds must flow from the right motives, not reputational ones.

As a consequence, in societies that consistently reward prosocial behaviour, people tend to internalise their motivations to help others. The best way to convince others that you want to help them and fix our broken world is to develop a genuine passion for doing so.

Such passions are never pure or extremely deep. They must be scaffolded by the right incentives, and they can disappear if incentives suddenly change—hence the many justice-championing revolutionaries who lose their love of humanity when they acquire power. But it is genuine and sincere nonetheless, and you can’t understand humanity’s progress over recent centuries without appreciating its reality—from the scientists and doctors who devoted their lives to understanding and combating disease, to the reformers who fought for social justice against slavery and oppression, to the entrepreneurs who created technologies that lifted countless people out of poverty.

Conclusion

The truth is messy and complex. We are rational creatures whose apparent “stupidity” is often a symptom of hidden strategies, but we are also maladapted to modernity’s vastness and complexity. In this strange new world, ignorance is our default, our intuitions mislead us, and the information environment exacerbates our confusion. To escape this bleak situation, we must unlearn our “common sense”. We need to be enlightened by insights and knowledge that only systematic, intellectual inquiry can provide.

Such inquiry has demonstrably improved the human condition. Liberal norms and institutions, products of hard-won, counterintuitive discoveries, function to channel our self-interest and ambition into cooperation and progress, helped along by a craving for prestige that can be—and has been—directed towards noble pursuits that have made the world measurably better.

And yet, at present, a shocking number of people are ignorant of this progress, and of the insights that underpinned it. If you read survey data or listen to the speeches of some of the West’s most popular politicians, you discover that many people sincerely believe that things have been getting worse.

This is a big misunderstanding.

To correct it, we must insist on the value of intellectual insights and carefully collected data. We must acknowledge that too many people are ignorant and confused about the world they inhabit, and celebrate those who aim to change that.

Further Reading:

• David Pinsof is one of my favourite writers and social scientists. I’m not sure how much he would ultimately disagree with my arguments here.

• On the role of counterintuitive liberal insights in the Enlightenment, I’ve been highly influenced by Joseph Heath, Jonathan Rauch, and Steven Pinker.

• On how status competition can be (and has been) channelled into cooperation and progress, see Will Storr and Nichola Raihani.

• On social complexity, political modernity, and evolutionary mismatch, see Walter Lippmann and Pascal Boyer.

• On why people are sincerely misled about the fact of modern progress, see Max Roser and Hannah Ritchie.

• How prevalent is misinformation?

• What even is “misinformation”?

• Is there a difference between politics and science?

• How impactful are propaganda, influence campaigns, and advertising?

• What impact has social media had on modern democracies?

• How worried should we be about the impact of generative AI, including deepfakes, on the information environment?

• The “liar’s dividend”

• Whether ChatGPT is more accurate and less biased than the average politician, pundit, and voter.

Links

• Sacha Altay

• “Misinformation Reloaded? Fears about the Impact of Generative AI on Misinformation are Overblown” Felix M. Simon, Sacha Altay, & Hugo Mercier

• “Don’t Panic (Yet): Assessing the Evidence and Discourse Around Generative AI and Elections” Felix M. Simon & Sacha Altay

• “The Media Very Rarely Lies” Scott Alexander

• “How Dangerous is Misinformation?” Dan Williams

• “Scapegoating the Algorithm” Dan Williams

• “Is Social Media Destroying Democracy—Or Giving It To Us Good And Hard?” Dan Williams

• “Not Born Yesterday: The Science of Who We Trust and What We Believe” Hugo Mercier

• Joseph Uscinski

• “Durably Reducing Conspiracy Beliefs Through Dialogues with AI” Thomas H. Costello, Gordon Pennycook, & David G. Rand

• “The Levers of Political Persuasion with Conversational AI” Kobi Hackenburg, Ben M. Tappin, et al.

• Ben Tappin

Chapters

• 00:00 Understanding Misinformation: Definitions and Prevalence

• 04:22 The Complexity of Media Bias and Misinformation

• 14:40 Human Gullibility: Misconceptions and Realities

• 27:28 Selective Exposure and Demand for Misinformation

• 29:49 Political Advertising: Efficacy and Misconceptions

• 35:13 Social Media’s Role in Political Discourse

• 40:50 Evaluating the Impact of Social Media on Society

• 42:44 The Impact of Political Content on Social Media

• 46:57 The Changing Landscape of Political Voices

• 51:41 Generative AI and Its Implications for Misinformation

• 01:03:46 The Liar’s Dividend and Trust in Media

• 01:14:11 Personalization and the Role of Generative AI

Transcript

• Please note that this transcript was edited by AI and may contain mistakes.

Dan Williams: Okay, welcome back. I’m Dan Williams. I’m back with Henry Shevlin. And today we’re going to be talking about one of the most controversial, consequential topics in popular discourse, in academic research, and in politics, which is misinformation. So we’re going to be talking about how widespread is misinformation? Are we living through, as some people claim, a misinformation age, a post-truth era, an epistemic crisis?

How impactful is misinformation and more broadly domestic and foreign influence campaigns? What’s the role of social media platforms like TikTok, YouTube, like Facebook, like X when it comes to the information environment? Is social media a kind of technological wrecking ball which has smashed into democratic societies and created all sorts of havoc? And also what’s the impact of generative AI when it comes to the information environment?

Both when it comes to systems like ChatGPT, but also when it comes to deepfakes, use of generative AI to create hyper-realistic audio, video, and images. Fortunately, we’re joined by Sacha Altay, brilliant heterodox researcher in the misinformation space, who pushes back against what he perceives to be simplistic and alarmist takes concerning misinformation.

So we’re going to be picking Sacha’s brain and just more generally having a chat about misinformation, social media, and the information environment. So Sacha, maybe just to kick things off, in your estimation, if we’re keeping our focus on Western democracies, how prevalent is misinformation?

Sacha Altay: Hi guys, my pleasure to be here. So it’s a very difficult question because we need to define what is misinformation. So we’ll first stick to the empirical literature on misinformation and look at the scientific estimates of misinformation. For that, there are basically two ways or three ways to define misinformation. One of them is to look at fact-checked false news.

So false news that have been fact-checked by fact-checkers as being false or misleading. And by this account, misinformation is quite small on social media, like Facebook or Twitter. It’s in between 1 and 5% of all the content or all the news that people come across. So according to this definition, it’s quite small. There is some variability across country. For instance, it seems to be higher in country like, I don’t know, the US or France than the UK or Germany.

There is another definition which is a bit more expansive because the problem with fact-checked false news is that you rest entirely on the work of fact-checkers and of course fact-checkers cannot fact-check everything and not all misinformation is news. So you see the problems. So another way is to just look at the sources of information and you classify them based on how good they are and how basically how much they share reliable information, how much they have good journalistic practice, et cetera. And the advantage of this technique is that you can have a much broader range because you can have, I don’t know, 3,000 sources of information that share information. And usually it broadly like most of the information that people see. And according to the definitions, misinformation is also quite small. So the definition is just for misleading information that comes from the sources that are judged as unreliable. And by this definition, misinformation is also quite small. Again, it’s like about like one to 5% of all the news that people encounter.

But then of course, the problem is not all the information that people encounter comes in this form. And for instance, some of it can come in terms of like images or all the sorts of things. And so this broadens the definition of misinformation. So some people think that when you broaden this definition, you have much more misinformation. My reading is that when you broaden this definition, you actually include so much more information that you increase the denominator. So of course, there’s going to be more misinformation, but because the denominator is larger, the proportion is going to be pretty much the same. But that’s an empirical question. So let’s say to sum up that it’s smaller than people think, according to the scientific estimates.

Henry Shevlin: If I can just come in here, a point that Dan you’ve emphasized in our conversations to me, and I think Scott Alexander has also emphasized in a great blog post called The Media Very Rarely Lies, is that a lot of what people think of as misinformation is just true information selectively expressed or couched in a way that naturally leads people to maybe form false beliefs but doesn’t involve presentation of falsehoods. Does that sort of feature in any of these sort of more expansive definitions of misinformation? Is it possible to create definitions that can capture this kind of deceptive, intentionally deceptive but not strictly false content?

Sacha Altay: I’d say that when you look at the definitions based on the sources, if a source is systematically biased and systematically misrepresent evidence and stuff, they are going to be classified as misinformation. I think the problem and the more subtle point is that these sources are not very important because people don’t trust them very much. But the bigger problem is when much more trusted sources who have a much larger reach, like I don’t know the BBC or the New York Times, they are accurate like most of the time, but sometimes and on systematic issues like I don’t know, they can be wrong. And that’s the bigger issue because they are right most of the time. So they have a big reach, they have big trust, but they are wrong sometimes. And that’s the problem.

Dan Williams: But I think just to focus on that observation of Henry’s, you might say, well, they’re accurate most of the time, but nevertheless, you can have a media outlet which is strictly speaking accurate most of the time with every single news story that it reports on. But because of the ways in which it selects, omits, frames, packages, contextualizes information, nevertheless end up misinforming audiences, even if every single story that they’re reporting on is on its merits, sort of factual and evidence-based.

I mean, I think the way that I understand what’s happening in this broader debate about the prevalence of misinformation is round about 2016 when we had Brexit in the United Kingdom and then the first election of Donald Trump, there was this massive panic about misinformation because many people thought maybe that’s what’s driving a lot of this support for what gets called like right-wing authoritarian populist politics. And around that time when people were thinking of the term misinformation, they were kind of thinking of fake news in the sort of literal sense of that term. So false outright fabricated information presented in the format of news. And as you pointed out, when researchers then looked at the prevalence of that kind of content, which you don’t really find when it comes to establishment news media for the most part, like there are always gonna be exceptions, that stuff is pretty rare.

And then one of the responses to that is to say, okay, if you’re only looking at like outright fake news, then you’re missing all of these other ways in which communication can be misleading by being selective, by omitting relevant context through framing, through kind of subtle ideological biases.

And then my view on that is, well, once you’ve expanded the term to that extent, and you’ve got this really kind of elastic, amorphous definition, it becomes really kind of analytically useless. Like you’re just bundling together so many different things. And that kind of content is also really pervasive in my view, within many of our establishment institutions, including within the social sciences. But Sacha, it sounds like you don’t necessarily want to endorse that last point. You seem to be thinking, even if you do have this kind of very broad definition of misinformation, we can still say that it’s a pretty fringe or pretty rare feature of the information environment. Is that fair? Am I understanding you right? Or is there something different going on?

Sacha Altay: I think I would agree with you that if the simple fact of framing information or having an opinion, like any scientist, even in the hard sciences, they have some theories that they prefer, they are more familiar with certain frameworks, and so they are going to be biased anyways. Scientists are humans, they are biased, but calling physics or the theory of relativity or whatever misinformation because it omits certain facts that it cannot accommodate or whatever, I think it’s far-fetched. I think it goes too far. So yeah, I would agree that if you use this broad definition of misinformation, then it’s very widespread. But then, yeah, even theories in physics would be misinformation because they cannot be completely objective.

I think science works not because scientific individuals are perfect, etc., or even because one theory is perfect, but because as a whole and as an exercise of arguing, etc., we get better and a little bit closer to the truth. But still, we are not getting at the truth and we cannot avoid the mistakes that you’re pointing.

Henry Shevlin: If I just want to push back a tiny bit, it seems to me, so obviously there’s this point here that, you know, all theory is value laden, the kind of physics point that I think is maybe true, but not very interesting. But I think there is maybe something in the middle here that is what I worry about, which is cases where there might be really quite, quite deliberate pushing of an agenda, a realization by a media provider that they are generating maybe inaccurate views, but they’re doing so just through reporting factual things.

So one example, Dan, that you’ve given before is that most of the kind of what we think of as misleading anti-vax discussion just reports on true factually accurate but rare vaccine deaths, but just reports on them in a very regular fashion. In the same way, you might think that selective reporting of certain kinds of violent incidents, whether it’s terrorism, police shootings, leads systematically to overestimation of the incidence of this kind of phenomena by the public or increased worries about its prevalence in a way that I think is perhaps worrying and politically objectionable, right? I think we might say, hang on, it is bad that we give so much press coverage to event type X rather than event type Y. And we know that this leads the public to overestimate the prevalence of event type X compared to event type Y. So I think there’s something in between the sort of, well, even physics is biased and the view of misinformation as, you know, strictly speaking lies. This kind of third category. I don’t know if that, I defer to you both as misinformation experts, but it seems that that is a worrying category.

Sacha Altay: I think you’re totally correct. And that’s what the field of misinformation has been proposing, like just for instance, classifying headlines based not on whether they are true or false, but whether they will create misperceptions after you have read them. And so researchers are saying, for instance, that we should classify as misinformation headlines such as, “a doctor died a week after getting vaccinated and we are investigating the cause.” And I think I disagree with this. I disagree with this thing that we should classify this.

What you were suggesting, Henry, was a bit different, is that it needs to be systematic. If you systematically misrepresent vaccine side effects, then it becomes problematic. But reporting on vaccine side effects and their possible negative effects is normal. And I think it’s healthy that news outlets are able to talk about and cover negative effects of vaccines, even if after reading the headlines, you have more negative opinions about vaccines, which is not supported by science, et cetera—they should be able to do that and they should do that. But if it’s systematic, as you say, I think it becomes more problematic. But I do think that when the bias is very strong in some of the definitions of misinformation based on the source, they would be classified as misinformation sources like Breitbart, et cetera. They are systematically extremely biased towards, I don’t know, these kind of things. And so they would be classified as misinformation.

Dan Williams: I think sort of one of the worries that I have though is who decides what constitutes systematic bias and bias about what? I think there’s a real kind of epistemological naivety that I often encounter with misinformation researchers where it’s like, you’re reporting accurate but unrepresentative events when it comes to vaccines. So we can call that misinformation. And then it’s like, well, as Henry mentioned, well, what about police killings of unarmed black citizens in the US. There’s a vast amount of media coverage of those sorts of events. Someone might argue that they are, statistically speaking, rare and unrepresentative, and that large segments of the public dramatically overestimate how pervasive those sorts of occurrences are.

And I think you go through many, many examples like that. And for me, the lesson to draw from that is not that, therefore, there are no differences in quality when it comes to the different media outlets in the information environment, like of course there are, but I also think like there’s such a thing as politics and there’s such a thing as science, where you’ve got scientists who attempt to acquire a kind of objective intellectual authority on certain things, and we should be very careful not to kind of blur the distinction between those two things.

I think when we’re talking about media bias in this really expansive way, where we’re not saying, okay, you’re just making shit up, but we’re saying you’re being selective in terms of which aspects of reality you’re choosing. For me, that’s a really important debate, but it’s a debate that happens within the context of politics and democratic debate and deliberation and argument. And I think sometimes I encounter misinformation researchers who treat that as if it’s just, it’s a simple sort of technocratic scientific question. Like we can quantify the degree to which the New York Times is biased or we can objectively evaluate the degree to which different kinds of outlets approximate the objective truth when it comes to their systematic coverage. And I get a little bit kind of squirmy when we get to that point, because I think that there’s just collapsing the distinction between kind of politics with all of its messiness and complexity and science, which I think should aspire to a kind of objectivity, which gets lost when we start making these really sort of expansive judgments.

I think we’ll probably circle back on this a few times as we go through this debate. But Sacha, you’re also somebody with very interesting views about not just this question of the kind of prevalence of misinformation, but also about human belief formation and the extent to which, in your view, lots of people, both in popular discourse, but also in academia, kind of overestimate the gullibility of human beings when it comes to exposure to false or misleading content. So do you want to say a little bit about your view concerning human gullibility?

Sacha Altay: Yeah, I just wanted to finish the last point on the fact that, you know, we are criticizing definitions of misinformation, but in media and communication studies for a long time, they have been studying kind of media bias, framing, agenda setting. Like they are very old theories of media, how they can misinform in subtle ways and indirect ways the public. And all of that has kind of been ignored by misinformation research. But now I feel like today misinformation research is catching up and be like, actually, we should go back to these theories. And so I think it’s good. But I just wanted to point that out.

And regarding gullibility, yes, I think it’s quite popular, the idea that people and like large complex events like the Brexit, Donald Trump or whatever are caused by people being irrational or gullible in particular. By gullible, I think what people often mean is that they are too quick to accept communicated information, like social information that they see out there in the world, in the news, communicated by others. And I think that the scientific literature shows something very different.

For instance, there is a whole literature on social learning, so how people learn either from their own experiences, their own beliefs, or what they see compared to like communicated information, social information, advice. And the consensus in this literature is that people underuse social information. They do not overuse it, they underuse it. And they would be better off doing many kinds of tasks if they were listening and weighing other people’s opinion and beliefs more than their own. So, I mean, it makes sense. Basically, we trust ourselves, we trust our intuitions, our experiences much more than that of others.

And so that’s kind of a consensus. There are many kinds of tasks, like you ask people, oh, what’s the distance between Paris and London? It’s like, 300 kilometers. Another participant, say 400. And you’re not going to take into account other people’s advice as much as your own intuition, even though you have no reason to be an expert on this kind of geographical distances. But you still trust yourself more.

And there are also many like theories and mechanisms that have been shown in political communication and media studies that I think suggest that people put a lot of weight on their own priors and their own attitudes when they evaluate and choose what to consume, which greatly reduces any kind of media effects or any kind of outside information. Like people are not randomly exposed to Fox News. They turn on the TV and they select Fox News. And then people selectively accept or reject the information they like the most. And so I think when you take all that into account, like selective exposure, selective acceptance, and egocentric discounting, it complicates a little bit the claim that humans are gullible.

Dan Williams: Yeah, so there’s this sort of popular picture of human beings as credulously accepting, you know, whatever content they stumble across on their TikTok feed. Although when I say human beings, it’s always other human beings, right? This is another point that you make with the third-person effect. Nobody really thinks of themselves as being gullible and easily influenced by false and misleading communication. But when it comes to other people, there’s this kind of intuition which is that, yeah, people are just being kind of brainwashed en masse by their lies and falsehoods and absurdities uttered by politicians and that they’re encountering in their media environment.

And your point is, no, actually, if you look at the empirical research, it doesn’t really support that at all. If anything, people put too much weight on their own kind of intuitions, their own priors, their own experientially grounded beliefs relative to the information that they’re getting from other people. So rather than thinking of many of our sort of epistemic problems as being downstream of gullibility, we should think of in some ways there being the opposite problem of people being too mistrustful, too kind of skeptical of the content that they’re coming across. Is that a fair summary of your perspective?

Sacha Altay: Couldn’t have said it better.

Henry Shevlin: If I can just raise one question here. Reading your brilliant paper, you emphasized, so this is a paper with the Knight Columbia School. You go through all these different misconceptions about how easily influenced people are by different sources, by sort of different peers, by the media, by the news. But this sort of does prompt the question, you know, where do people’s beliefs actually come from?

And you mentioned people’s priors, people’s intuitions, but presumably people aren’t born with these intuitions, they are formed from somewhere through certain kinds of processes. So I’m just curious if you have any sort of thoughts on where do people’s views come from? Because obviously that would suggest, well, that’s the place you go then if you want to influence people, you intervene on whatever is causing this fixation.

Sacha Altay: I mean, my view on beliefs, and I mean, much of my views come from Dan Sperber and Hugo Mercier, who have these theories on like reasoning and the roles of beliefs. And so basically, to answer your question, I think a lot of people’s beliefs are downstream of their incentives and intuitions they have about the world. For instance, vaccines. Vaccines are like profoundly counterintuitive. Like it’s very difficult intuitively to like vaccines. Like first there’s a needle that goes into your arm, there’s a little bit of blood, you think that there is some kind of like pathogens inside the vaccines, like it’s not something that’s very intuitive. So first I would say most, like not necessarily the beliefs, but the attitudes people have about vaccines largely comes from these very general intuitions that they have about contagion, about infections and about all these things.

And then the beliefs, well, people need beliefs to justify their attitudes. And so if your doctor is like, do you want to get vaccinated and you don’t really want to get vaccinated, you can say you’re scared of needles. But if there are also some widely available cultural justifications like, vaccines cause autism, maybe you’re going to jump on it. Maybe you’re not going to jump on it because maybe you’re smart and you know it’s false, et cetera. But you need justifications. And so I think a lot of people’s beliefs comes from this need to rationalize some justifications that they have. And I think that’s also why on many topics, people don’t have that many beliefs because often people don’t really need to justify many of their attitudes. And there’s a lot of work, for instance, in political science on how surveys kind of create beliefs in people because people have intuitions and kind of like vague opinions about all sorts of stuff. But when you ask them, they have to fix it and they have, and in some sense, it creates the beliefs.

So yeah, I would say beliefs mostly come from prior attitudes that people have and incentives that they have to act in the world.

Henry Shevlin: Okay, but those... just to push a little bit harder there, so the prior beliefs, I think we’re just still kicking the can down the road a little bit. Incentives I get. Incentives seem genuine and explanatory here, but presumably it’s not the case that you can predict people’s vaccine attitudes from the degree of phobia they have towards needles, right? Or at least, even if that is predictive, I don’t know if it is, it seems like there’s more going on there. I don’t want to give people, and I think that’s the danger of giving people too much credit for saying, oh, people’s beliefs perfectly track their own incentives. I can totally agree that incentives play a role, but I’m sure just when we think about our own sort of peer groups, right? I disagree with the political views of a lot of my peers, despite us being in the same socioeconomic class, despite us working in the same industry, despite us having, you know, broadly similar interests, I would have thought. So, I don’t know, I can see incentives carry us some of the way, but yeah, they don’t completely close the mystery here.

Sacha Altay: No, of course, of course. I think it’s, you take the example of vaccines. I think most people who get vaccinated, they just get vaccinated because they trust institutions, they trust their doctors. Maybe they have seen their doctors for 20 years, their doctors tell them to get vaccinated, they do it. So that’s the main explanatory agent here is just they trust some institutions, some experts who tell them to do something and they do it.

You wanna jump in, Dan?

Dan Williams: Yeah, I was just going to say, I think it seems like it’s possible to think, and as I understand Sacha, your view, this is your view. It’s possible to think that we overestimate the degree to which people are kind of influenced by whatever content they happen to stumble across in their media environment or the viewpoints that they happen to encounter in their social network—that we tend to think people are too gullible when it comes to those things.

It’s possible to think that, but also to accept that, of course, we are going to be influenced in complex ways by the information we get from people that we trust, from sources that we trust, from our upbringing, from our social reference networks and so on. So the idea that we’re not gullible and not credulous shouldn’t be sort of conflated with the idea that we somehow are born with our entire worldview from the start in ways that aren’t influenced by the media environment and by the testimony that we encounter. Like clearly we’re massively influenced by what we hear from other people, but sort of my understanding of the perspective that you’re outlining Sacha is that process whereby we build up beliefs about the world—firstly, there are some things that just everyone kind of finds natural, like maybe like there’s something weird about vaccines when you hear about the concept, most people just have a kind of instinctive aversion to it, but also things like, you know, my group is good, the other group is bad, or like certain kinds of maybe xenophobic tendencies that come naturally to people and so on. So there are certain ways of viewing the world and certain things which are intuitive, maybe as a consequence of our evolutionary history, and that interacts then in very kind of complex ways with our experiences, with our social identities, with our personality, with the people that we trust, the institutions that we trust, those we mistrust, and so on and so forth. So you can accept all of that and the role of social learning within that whilst also thinking people tend to exaggerate how gullible, how credulous people are when it comes to sort of incidental exposure to communication. Is that your view, Sacha? Is that a kind of accurate representation of it?

Sacha Altay: Yes, yes, yes it is. I think a lot of the reason, like when we change our mind drastically, it’s either because like we have a lot of reasons to trust the source. Like if the BBC says that the Queen died and the BBC says it, the Guardian says it, we’re going to update our beliefs immediately. And most people, even the people who distrust the BBC are going to update their beliefs directly.

And it’s the same if like, I don’t know, my wife tells me that there is no more milk in the fridge and I have to buy some. I’m going to update my beliefs about the milk in the fridge and buy some, you know, in some ways, of course we update our beliefs based on the information that’s provided to us. It’s just that we do so I think in ways that is broadly rational in the sense not that it’s perfect, but that it serves our everyday actions and our incentives, like what we want to do in the world, like very well. So I think that’s also the way in which I mean it is that when we update it and when we do it, we do it quite well, not to discover the truth, but at least to get along in the world.

Dan Williams: And could you maybe say a little bit more about this point concerning selective exposure? So the fact that when people are engaging with media, with the viewpoints of pundits and politicians and so on, a lot of that is, quote unquote, demand driven in the sense that people have strong attitudes, they’ve got strong political, cultural allegiances, they identify with a particular in-group, they want to demonize like those people over there or that kind of institution, et cetera. And it’s these sort of pre-existing attitudes, interests, allegiances, which often build up in complex ways over a long period of time, which then causes people to kind of seek out information and often misinformation, which is consistent with their attitudes and their interests, rather than the picture I think sometimes people have, which is—I think the way Joe Uscinski puts it is, you know, they’re walking along and they slip on a banana peel, you know, they encounter some conspiratorial content on social media and now they believe in QAnon or like Holocaust denial. That’s just not the way that it works. Could you say a little bit more about that concerning like selective exposure and the demand side of misinformation?

Sacha Altay: Yeah, we know for instance that misinformation on social media like Facebook or Twitter, which have been the most studied in particular Twitter, you have a very small percentage of individuals who account for most of the misinformation that is consumed and shared on these platforms. And it’s like very small. It’s like 1% or less than 1% that account for most of the misinformation that is consumed and shared on these platforms.

And these people, they are misinformation sharers and consumers, not because they have like special access to misinformation because they have a lot of money or whatever, but simply because they have some traits that make them more likely to seek out such content, such as having low trust in institutions, being politically polarized. And because of these traits, because they don’t trust institutions, they are looking for counter-narratives to like the mainstream narratives they find on mainstream media. Because the thing is that these people who consume and share most of the misinformation on social media, and give us the impression that there is a lot and that many people believe it—these people are also exposed to mainstream narratives. It’s just that they decide to reject the mainstream narratives and instead of trusting what the TV tells them, they go on some Telegram channels, they go on some weird websites to learn about the world and do their own research.

And this is, I think, some of the strongest evidence, at least in the case of misinformation, that the problem is not in the offer of misinformation because it’s actually quite easy, quite free, quite accessible. It’s super easy to find misinformation online, but most people consume very little of it. But you have a small group of people who are very active and very vocal who consume most of it. And they have low trust in institution and are highly polarized. And I think it matters a lot for how we want to tackle the problem of misinformation. The problem is not that you have a majority of the population that’s kind of gullible and so we should avoid them being exposed to misinformation, rather you have some people who have some very strong motivations to do some specific stuff. And I think we should address these motivations. And because addressing the offer is impossible. And I’m not against like content moderation and stuff. I think we should try to be in an information environment where the quality of the information is the highest possible, et cetera. But if you have motivations to look and to pay or to consume some content, then the offer will be met, like people will create such content.

Dan Williams: Could we maybe just before we move on to these issues about kind of social media and AI, because I really want to get to those, there’s another point connected to this issue about gullibility where I think there’s this massive kind of gap between common sense, conventional wisdom, and what the empirical research shows, which you’ve written a lot about, which is like the impact of things like political influence campaigns and commercial advertising and so on. So you go into that in your paper on generative AI and why you think there’s been a lot of unfounded alarmism about that, which we’re going to get to shortly. But even separate from the issue concerning AI, could you say something about what the evidence that we have actually shows when it comes to the impact of political and sort of economic advertising campaigns?

Sacha Altay: So political scientists have been studying that for a while because in the US there is so much money that is being spent on political advertising, especially in presidential elections. And so the best studies, they come from political science. And to give you an example, some of them have up to 2 million participants that are being exposed to hundreds or thousands of ads for long periods of time, like months.

And so these are the kinds of study that are being done in this field, like very large sample, long periods of time, et cetera. And the consensus is that political advertising in presidential elections in the US has very, very small effects. The effects are not zero because of course, with such big sample size, long periods of time, et cetera, you do find significant effects, but the effects are very, very small, like point of percentage. And so that’s the consensus in political science in the US.

So it’s a bit specific because the US you have like Democrats and Republicans and you socialize these identities and these identities are very hard to change. Like if you’re a Democrat, it’s very hard for you to change and vote Republican. And of course, in the US, you often have only two candidates that are very prominent and people hear about them all the time. So it’s difficult to move the needle. But in like other elections in other country, multiparty, you have more room for political advertisement to have an effect. But even in these cases, even when it’s like lower stakes campaigns with less known candidates, the effects are still quite small. Like, I don’t know why we have this idea that advertisement works very well, it influences people, but at least when it comes to political voting, it’s just very hard to influence people’s vote. And it’s the same for like marketing, like online ads, like on social media—are very ineffective, the thing is that they are very cheap as well. So I don’t want to say that they are useless because they’re actually extremely cheap. So that’s why these companies do them a lot, but they’re also extremely ineffective. And so that’s the consensus in political science.

Henry Shevlin: So I had a question about this in relation to your paper again. It really paints quite a dismal view of the power of advertising in general. And yet this is like a vast global industry. Is it all just founded on sand? Is it all just smoke and mirrors? Are people basically wasting hundreds of billions of dollars a year on advertising that doesn’t, largely doesn’t work?

Sacha Altay: That’s the opinion of many people. Yeah. Many people think that at least it’s overblown. I don’t want to say that it’s completely useless, et cetera. Like, of course, if you want to buy like a washing machine, they all look the same. And if they are all about the same price, if you have more information about one and the information is good and the reviews are good, et cetera, you’re probably going to buy it more. But it’s just you already want to buy the washing machine and you have a price range and you have already like so at the margin, advertisement can work and has an effect, it’s just that the effect, like they calculate basically the elasticity. So how much more when you spend on advertisement, how much more will you sell basically? And the elasticity is like super small. It’s like, I forgot exactly, but it’s like very small.

But yeah, some people have written books about how the whole internet and know, products on the internet, like social media, et cetera, are free because we are the product and they sell us advertisement and stuff. And all of that is a bubble. Some people think that it’s completely a bubble. I don’t think it’s completely a bubble, but clearly I think, yeah, it’s overvalued. I think ads are a little bit overvalued. And I don’t think AI is gonna change that much.

Dan Williams: Okay, so just to sort of summarize what we’ve got to so far. So on this question of how prevalent misinformation is, if you’re focusing on fake news, it doesn’t seem to be anywhere near as widespread as many people think it is. Once you start stretching and expanding that definition to encompass more and more things, yes, misinformation so defined is much more widespread and plausibly is much more impactful, but it becomes so kind of amorphous that it’s difficult to apply scientifically.

Then the second thing we talked about was this issue concerning gullibility, where in your view, Sacha, and I agree with you, even though obviously people are influenced by social learning and there is evidence that, you know, persuasion can work, it can influence what people believe, people also tend to dramatically overestimate how gullible people are.

Let’s now turn to technology and where AI is relevant. And let’s start with social media, kind of very broadly construed. Henry, actually, why don’t I bring you in here? Because I think in a few of our previous conversations, you said something like the following, and you can tell me whether I’m remembering correctly. You said, we can contrast two kinds of cases, like video games and social media. In both cases, there was this big societal panic. Video games are going to make people really violent. They’re going to play Call of Duty, and then they’re going to go out and start shooting people in their community.

And your view is, the evidence there is actually incredibly weak and that there’s very little to support that kind of panic. Whereas when it comes to social media, there was a lot of panic, maybe not initially, actually, I think there was a lot of optimism about social media initially. But these days, there’s a lot of kind of concern about social media and how it’s, you know, destroyed democracy and human civilization itself. It’s this awful thing, having all of this sort of awful set of political consequences. And am I right, Henry, in thinking you’re actually quite sympathetic to that view about social media, even though you’re not sympathetic to the violent video games story.

Henry Shevlin: Yeah, yeah, no, great. I’m glad you bring up this example. Two things. One is I think my main point with that example is about sort of the time course of these worries that with violent video games, we had this massive initial panic that sort of died down as the evidence sort of basically didn’t arrive. As we saw that there wasn’t that as much concern as initially there was we thought there was reason to think there was. Whereas in the case of social media, there really wasn’t that much concern at first. It was seen as, if anything, a positive technology and concern has just sort of grown over time. And that sort of point about the time course of sort of the moral panic is sort of separate from the degree to which these are robust.

That said, I do, I am more sympathetic to the idea that social media presents an array of worries. So I’m probably more sympathetic than both of you to sort of Jonathan Haidt’s worries about the impact of social media and mobile phones on teenage mental health, which is a separate point from misinformation. I also worry about the role of social media and things like political polarization. Again, at least a little bit distinguishable from misinformation. But yeah, I guess I’m a little, at least a little bit worried about the role of social media and misinformation as well.

Dan Williams: Okay, I’ve got sort of views that are difficult to summarize about this. Let’s stay away from the teen mental health, because I think that opens up a whole can of worms, et cetera. Let’s focus on kind of the political impacts of social media broadly construed. Sasha, my understanding of your view is you basically think that the panic over social media and its political impacts is unfounded and it’s not well supported by evidence. Is that fair? Care to elaborate?

Sacha Altay: Yeah. So I’m just going to start by mentioning, I think, the scientific literature and what I think is the best evidence that social media have weaker effects than people think. So there have been many Facebook deactivation studies. So basically, you pay some participants to stop using Facebook for a few weeks. And in the control group, the participants are the same, but they are paid either to stop it for one day or to do something else.

And in general, what these studies find is that when you stop using Facebook for a few weeks, you become slightly less informed about the news and current events, suggesting that using Facebook regularly helps you slightly know about the world and what’s going on in the news. But it also makes you slightly more sad. So you’re slightly less happy when you use social media. So participants who deactivate social media, especially Facebook for a few weeks, are slightly happier. It’s not exactly clear why. It could also be because they are less exposed to news and news is sad and makes people less happy, etc. So it could be that. And there are also many other studies on Instagram.

And basically what all these studies suggest is that the effect of social media on stuff like affective polarization, political attitudes, voting behaviors, is either extremely small or no. And so the effects are very small. But now that I’ve mentioned this literature, I want to mention that there are many critics of this literature and of these experimental designs. For instance, even the longest RCTs are like two months. And of course, two months is super small at the scale of social media. They have been here for years. And you could imagine that it takes a few years for the effects of social media to kick in.

You can also imagine that, of course, participants stop using social media for a few months, but the world continues using social media. People around them continue using social media. So you kind of have these network effects that are possible. And of course, the effects of social media are not individual, they are collective. And so these RCTs are kind of missing the point. They cannot capture the collective and more systemic effects that social media could have. So that’s another critique. And there are many other critiques.

But I still think that what these RCTs show is that social media probably has effects. And there are studies like in collaboration with Meta showing that if you change Facebook or Instagram with like a chronological feed, that is instead of showing users the most engaging content, you show them the most recent content. When you do that, they spend much less time on the platform. Like the time they spend on the platform is diminished by one third.

And it has a lot of effects on in-platform behaviors, but very few effects on out-platform behaviors, on attitudes, on et cetera. So we should take these studies with a grain of salt, but I still think they show us that the effects are probably not as big as at least the most alarmist texts suggest.

Dan Williams: Hmm. I think maybe another critique that some people have raised is that these studies, especially that set of Facebook, Instagram studies that you mentioned, were conducted after there had been a lot of adjustments to the platforms and the algorithms in light of concern about things like misinformation and their effect on polarization and so on.

So that just goes to say, as you say, many people have generated lots of different criticisms of what we can really infer from these studies. I mean, my own view is they tell us something, which is that the most simplistic, alarmist stories about social media don’t seem to be supported by the current state of really kind high-quality empirical research. I don’t think they provide very strong evidence that should cause someone who goes into this with a really strong prior that social media is having all of these catastrophic consequences to update that much. And that then suggests that like how you view this topic is going to be shaped by a lot more than just the empirical research itself. So in your case, I assume that you’ve got these general priors about how media doesn’t have like huge effects on people’s attitudes and behaviors and these things are shaped by all sorts of complex factors other than media. And am I right in thinking that’s doing a lot of the work when it comes to your skeptical assessment over and above these studies themselves?

Sacha Altay: Yes, but I would say the strongest argument maybe in favor of my position is descriptive data on what people do on social media and how often they encounter political content. Because to be politically polarized, you need to be exposed to political content. And there are more and more descriptive studies, some of them on the whole US population in the US, showing that it’s less than 3% of all the things that people see on social media.

So less than 3% of all people see on Facebook is either political or civic content. And there are also super nice recent studies that are using a novel methodology, which is basically recording what people see on their phones. So it’s like a lot of participants download an app and the app records what people see on their phones like every two seconds or so. And these studies have shown that in the last US presidential election, for instance, people have been exposed to content about like Donald Trump less than three seconds per day. So during the US presidential election people have seen so little political content on their smartphones that it’s ridiculous and it’s so small that in my opinion it can only have small effects.

Then again a contrary argument could be it’s the average and they do find that you have a small minority who is exposed to a lot of political information but then again who are these people? Again I think they have attitudes, have priors and they have motivations, they are partisans. And yes, misinformation or content on social media can reinforce, exacerbate, radicalize them a little bit. But I think for the mass and for the general public, who’s generally not that interested in politics, etc. I don’t think it can have very strong effects.

Dan Williams: Yeah, I just want to double click on that and then I’ll bring Henry in. One other kind of stylized fact, which we should flag, which I think is surprising to some people, is if you’re the kind of person who cares about politics and follows the news carefully, and you read political commentary and so on, you are extremely unrepresentative of the average person. Most people don’t follow politics. They don’t follow current affairs closely at all.

And if you ask people very, very basic questions about politics, they are shockingly uninformed about things. That is shocking relative to the perspective of someone like us who follows politics very, very closely. And that’s another thing which I think people who are highly kind of politically engaged often get wrong when they’re thinking about this topic. If the picture in your head when you’re thinking about social media and politics and so on is that the person who’s constantly posting on X about politics is representative of ordinary people. You’ve got an incredibly skewed, misleading picture.

Okay, there’s tons, I think, more to say here. Henry, did you want to come in with any kind of pushback or any more articulation of your perspective?

Henry Shevlin: Yeah, this is all really interesting and helpful. I guess the only thing I’d say is that it seems to me social media has also just changed the kinds of voices that get platformed in the first place in a way that’s both positive and negative. But, we think about things like the rise of Tumblr and its contribution to sort of a lot of so-called, you know, woke discourse, particularly in sort of the late 2010s. And we could equally say the same thing about, for example, reactionary bloggers or neo-reactionary bloggers like Curtis Yarvin and so forth. I think these are the kind of voices that probably just wouldn’t have found an outlet in the prior social media ecosystems. Maybe that doesn’t matter, right? If none of this stuff actually impacts people’s views that much. But it does seem like an interesting shift in our broader political media landscape that social media has changed not just the kind of how much time people spend interacting with content or the way in which they do so, but also the kind of content that gets out there in the first place. Does that figure at all in the impact of these things?

Dan Williams: Sacha, before I bring you on, just want to say just one really quick thing about that, which is the reference to Curtis Yarvin there made me laugh because I think like he’s an example where like the overwhelming majority of people won’t be aware of him. But I think he probably is influential within the kind of ideas, intelligentsia space of the political right. But this idea that like social media and the affordances and incentives of social media kind of changes which voices become influential and prestigious. I think that’s such an interesting and important point, but Henry, I thought you were going in the direction of, like someone like Donald Trump can absolutely murder on social media because he’s so good at like tapping into the attention economy dynamics on social media in a way that, you know, he’ll be much less successful if we lived in a kind of Walter Cronkite kind of media environment.

But there’s this other aspect, which is like the decline of elite gatekeeping, which is characteristic of social media and it’s via that route, I think, where people like Curtis Yarvin can enter the conversation in a way in which they probably wouldn’t have been able to if you go back to like the 90s, 2000s. Sorry, I just had to double click and say that. Sacha, did you want to respond to Henry’s point?

Sacha Altay: No, yeah, I agree. I just also want to say we often mention Trump as like the example of like someone we don’t like who benefits from social media, but there are also people who we like who benefit from social media like Barack Obama. Like he used Facebook a lot during his campaign. He’s super charismatic. And if he was president today or if he was running today, he would do great on TikTok. He still does great on TikTok. Like he’s so charismatic, so good. So it doesn’t always benefit the worst actors.

And I want to say, it’s a very important point about how social media may also shape how politicians communicate. There are some studies, for instance, in France on how short format videos like TikTok is changing how parliamentary members are talking at the parliament. And there are studies showing that especially at the extreme and especially the extreme right, they are doing more and more speeches that are like with more emotions and more, I don’t know, buzzwords. And what they say is that then they post this on social media, and the more buzzwords, the more emotions, and the more all of that, the more it’s going to go viral. And so that their goal is not to convince other parliamentary members, but instead just to buzz on social media and reach some parts of the population.

Then it’s a normative question, whether it’s good or bad. Probably using emotions and stuff is bad. But you could also imagine that if they were speaking to the general public in more authentic way and try to reach them because a lot of people are not interested. That could also be good. But of course, because it’s the extreme right and stuff, we don’t like them. And I think we have good reasons not to like them. But I think we should be careful and we should also think of ways in which it could be used to do good stuff. But I agree that in general, it probably hasn’t done very good. And it’s very hard to quantify it.

Dan Williams: Just before we move on to the topic of generative AI, my view is there’s so much uncertainty in this domain when we’re asking these really broad questions like what’s the impact of social media on politics that we can’t really be very confident about any view that we might have. But it does seem like, at least in my view, a lot of the popular discourse and academic research has focused on things like recommendation algorithms and filter bubbles and so on, where I think I’m very close to your view, Sacha, in thinking that there’s just a lot of kind of unfounded alarmism. But there’s this other aspect of social media, which I think probably has been very consequential, which is just its democratizing consequence. The fact that like prior to the emergence of social media, it was a much more elitist media environment. Whereas now, anyone with a phone, a laptop, whatever, can open up a TikTok account, get on X and start posting about their views.

And I don’t think you need to view that through the lens of, well, that means they’re going to start articulating their views and then persuading large numbers of people. But what I think it does is certain views, which were kind of systematically excluded from the media environment before the emergence of social media, can now become much more normalized. And also people can achieve kind of common knowledge that other people share views that used to be much more marginalized and stigmatized. So those sorts of views can end up being more consequential in politics, even though the views themselves aren’t necessarily more widespread.

And I think you find that with things like conspiracy theories. My understanding of the empirical research, again, people like Joe Uscinski, is that the actual number of people who endorse conspiracy theories hasn’t really increased, but they do seem to play a more consequential role within politics because people with really weird conspiratorial views used to be kind of marginalized in media. Whereas now it’s very easy for them just to start expressing those views online, finding people who share similar kinds of views, coordinating with them. And so it can play a bigger role in politics, even though it’s nothing to do with, you know, mass algorithmically mediated brainwashing or anything like that.

Okay, I’m sort of conscious of time and I really want to focus on generative AI. So there was this big panic about how once we’ve got deepfakes and other features of generative AI, this was going to have really disastrous consequences on elections. It’s going to shift people’s voting intentions in all sorts of dangerous ways. Sacha, you’ve written a paper which we’ve already referred to with Felix Simon. Looking into the evidence on this and presenting a kind of framework for thinking about it, what’s your take?

Sacha Altay: I will start by saying there are three main arguments why people are worried about the effect of generative AI on the information environment. The first one is that generative AI will make it easier to create misinformation and basically to kind of flood the zone with misinformation. The second one is that it will increase the quality of misinformation, better, faster misinformation. And the last one is personalization.

Generative AI will facilitate the personalization of misinformation. I think these are the three main ones and I can go quickly over them and argue why I don’t think they are a big deal. So about quantity, I think that quantity does not really matter. Today we are exposed already to so, I mean, there’s already so much information online and we are exposed to a very tiny fraction of that information. So adding more content does not necessarily mean that people will be more exposed to it. And I think it’s particularly true in the case of misinformation, where I think demand plays a very important role. And so it’s not because there is more misinformation that people will necessarily consume more misinformation. Like it’s not because you have more umbrellas in your store that people will buy more umbrellas. There needs to be like factors, like I don’t know, rain. If it rains more, you will sell more umbrellas. But so there needs to be something, there needs to be like incentives for people to demand more misinformation, to consume more of it. And that’s why I don’t think that the quantity argument is very strong.

I think also the cost of producing misinformation are already extremely low. Like we see it with Donald Trump or whatever, they just say something that is false and they say it with confidence and that’s it. Like the costs are very low. Also, we are very imaginative as a species, like humans have come up with like incredible, fascinating, engaging stories. And of course, AI can improve our innovative skills, but still we are very good at making up stories that make us look good, that make our group look good. And so I don’t think generative AI is going to help that much in creating more misinformation. Regarding quality, yeah...

Dan Williams: Just to interrupt you so that we can sort of take these step by step. So the first worry is generative AI both with kind of large language models and the production of text, but also deepfakes. I take it you’re including kind of both of those categories. The worry is, well, this is going to just really reduce the costs of producing misinformation. Therefore you’ll get this explosion and the quantity of misinformation and that’s going to produce all sorts of negative consequences. And your view is, well, the bottleneck that matters isn’t really quantity anyway. It’s like what people are paying attention to. So you can increase as much misinformation, you can increase the amount of misinformation as much as you want. And in and of itself, that’s unlikely to have a big impact on people’s attitudes and behaviors. Do you have any thoughts about that, Henry, before we move on?

Henry Shevlin: I guess one concern would be even though media environments are flooded with content already, and I completely agree attention is the sparse commodity, maybe you could think of generative media as allowing sort of very niche areas to get flooded with content in a way that wouldn’t have been easy before. I’m just thinking here’s a silly little example, maybe an interesting example from recent media. Some of you may have seen the anti-radicalization game that was launched in the UK, featured this character about two weeks ago, featured this character called Amelia, a purple-haired anti-immigration activist in this fictional game, which was quickly seized upon by a lot of the anti-immigration right in the UK. And now there’s a flood of AI-generated content all about Amelia, mostly making her look really cool and some of it kind of playful, some of it kind of silly. But the point is this was just like a niche news story that I think people found amusing, but I think it would have died a lot quicker had it not been for the ability of people to seize upon this and generate huge swathes of content about Amelia in a very, very short time. So maybe there was just pre-existing demand there, but it would have been demand that would have been perhaps hard to meet without the ability of generative AI tools to create the content to meet that, which maybe is a difference.

Sacha Altay: Yeah, no, I mean, that’s possible. But when you look at the memes on the internet, most of them are like very cheap. It’s just like an image with some text and you just change a little bit the text and we’re probably going to go into that. But it’s the same with like deepfakes. Like cheapfakes are much more popular than deepfakes because they are super easy to do. Like you just change the date or change the location of something and boom you have your cheapfake. And that’s why they are super popular. Yeah, I don’t know, anyway.

Dan Williams: What’s the definition of a cheapfake, Sacha?

Sacha Altay: A cheapfake is just a low-tech manipulation of information. Like you have an image and you change the date of the image, or you change the location of the image. So in opposition to deepfakes, which are like high-tech, completely like for instance, generated image where like it’s usually sophisticated, et cetera. Cheapfakes in opposition, like very cheap, like that you can, most people can do with their computer without like requiring any tech skills basically.

Dan Williams: Sorry, I think I cut you off. I just wanted to give some clarity to people who weren’t familiar with that. Okay, so that’s quantity. And the next thing that you mentioned as a sort of worry that many people have is quality. That is generative AI won’t just enable us to increase the amount of misinformation but increase its quality and initially at least you’re understanding quality as being different from personalization. You’re treating that separately, is that right? Okay, so give us like, surely the concern here is just that, okay, quantity in and of itself isn’t gonna make a difference. But once we’ve got the capacity to generate like incredibly persuasive text-based arguments and deepfakes, even if it’s true that you can create these sorts of cheapfakes and they can be influential, in different contexts, surely the quality of the misinformation must make a big difference to how many people get persuaded by it.

Sacha Altay: Yeah, I think quality is the most perhaps intuitive argument because it’s the idea that you’re going to be able to create images or videos that are unrecognizable from real videos or images. And so of course people are like, how am I going to trust images or videos anymore if they are unrecognizable from real ones? So I think that’s like a very fundamental fear that people have. And I think it makes a lot of sense. It’s very intuitive. But I don’t find it very convincing.

I think it raises a lot of challenges, but I don’t think it raises enough challenges to be alarming. For instance, I think we have had this challenge before with photography. We have been able to manipulate photography in ways that we cannot distinguish them from real photography since the beginning of photography. And how did we solve this problem? Not with technical tools or whatever, but just with social norms about the use of images to not mislead others.

And we have been able to create like fake texts or say false stuff forever and we haven’t solved the problem with like some fancy tech innovation but simply by having rules, reputation, social norms and trusting more or less people based on what they have said before based on what our friends think of them based on their past accuracy and I think all of this we will still be able to use it to help us navigate an environment in which videos could be AI generated or could be real.

And I mean, something I’ve mentioned before, but quite fundamental is that, for instance, we trust the BBC or the New York Times to be broadly accurate most of the time. We also trust them to not use AI in misleading ways and not share like deepfake footages of like presidential candidates that mislead us. And I think this trust and the institutions that exist are sufficient to prevent most of the harm from this.

I think this will have effects. For instance, maybe we will be less able to trust people and sources that we don’t know. Because if we don’t have their track records, how can we trust them that the information they are sharing is true or false or AI generated or not? But I think that’s a very old problem and we will manage. It will make it more complex, but I think we’ll manage. Yeah, Henry?

Henry Shevlin: I was going to say though, isn’t there a worry that sort of new technology creates kind of normative gaps that allow for sort of a kind of annealing or a kind of recalibration of norms? I’m thinking about something here like file sharing, for example. Like I’m of the generation where, you know, Napster, the generation where suddenly it became possible to download music for free. And this created a whole bunch, a whole shift in norms where I think for my generation, at least, you know, this form of theft was basically just completely normalized. Hence we had advertising campaigns like you wouldn’t steal a car, therefore why would you download a song or a movie? And basically pirating went from something that was niche and maybe frowned upon to something that was just completely normalized.

In the same way, I think you might worry that the ease, ubiquity of generative AI is gonna shift our norms around creating fake content. And arguably we’re already seeing this. We had just a very recently the White House itself retweeting pictures, I think of a protestor at an anti-ICE rally and they had manipulated the image, right? And you know, I think if called out on that, they’d probably say, yeah, sure, you know, of course, yeah, we play around with images, you know, that’s what generative AI can do. That’s just the way things work these days, which does seem like a normative shift perhaps, one partly occasioned by technology.

Sacha Altay: My intuition is quite the opposite, is that if anything, challenges that AI, these new challenges of AI will instead increase the epistemic norms that we have. And because we want to know the truth, like we don’t want to be biased. We don’t want to be misled. We don’t want to be misinformed. And so the fact that the challenge is becoming harder, that it’s going to become harder to know if a video is true, authentic or not, is going to make us harder and harsher on people who do as the White House did, where they, we don’t know if it’s them who manipulated it or not, but they share the manipulated images that do not portray her accurately. And so I think people are going to be angry at that. And I think it’s just going to increase how people, the level of the expectation, like what people expect. And I think people are going to expect more. They’re going to expect news outlets and people to be better. I mean, it’s just a prediction. I hope I’m right. I’m an optimist, but...

Dan Williams: So can we connect that to the worry many people have about the liar’s dividend? The idea that, once we’ve got deepfakes, I mean, we’ve currently got technology to create hyper-realistic audio and video recordings, which are basically indistinguishable from reality. There’s the kind of initial worry many people have, which is, my God, people are going to become persuaded en masse that this stuff is true. And I think that’s very unsophisticated as a worry.

But then there’s another story people have which is, okay, maybe it won’t persuade people, but now that you’ve got the capacity to create these deepfakes, politicians, elites, other people who do shady things, they can use the possibility of something being a deepfake to just dismiss any kind of recording which is raised against them as evidence of them doing something shady.

And I guess connected to that as well, there’s the worry people have, which is that just as consumers of content, now if we encounter any kind of audio or video which goes against what we want to believe, we can just say, well, it’s a deepfake. I don’t have to believe it. So we’re just gonna end up becoming like more and more cocooned within our own belief system, not having this access to learn about the world via recordings. So it’s a kind of liar’s dividend worry and this general worry that this is just going to just obliterate the kind of epistemic value, the informational value of recordings. What’s your thought about those kinds of worries?

Sacha Altay: First thing, I think the liar’s dividend does not hinge on AI itself, but rather the willingness of politicians and some elites in particular to lie and to evade accountability and responsibility. AI will certainly be a new weapon in the arsenal and we have seen it in the elections in 2024, etc. Many politicians have used AI to their benefit and many politicians and elites are continuing using them. So for sure, it’s something we should be, we should worry about and we should regulate, et cetera. But will it be a particularly good weapon in the arsenal? Will it be a game changer? I’m not sure. I mean, time will tell. So far, I don’t think it has been particularly good. I don’t think it has been used in particularly good ways. I don’t think people particularly buy it. I don’t think when people share something and then they’re like, no, it was just AI or like try to use AI as the excuse. I don’t think it works very well. And I think there are going to be reputational costs for people who try to do that. We are going to remember that they have tried to do that. And so I don’t know. Again, time will tell. It’s an empirical question. I may be wrong. I don’t know. Yeah, Henry.

Henry Shevlin: I was just going to chime in. I’m sure I’m not alone in having seen on Facebook in particular, lots of cases of AI-generated media being mistaken. I don’t want to pick on boomers too much, but it is often boomers who completely seem to buy it. Like you might have seen these examples of people breaking these glass bridges, these videos that went viral and lots of people, particularly I say older respondents who completely seem to believe this is a real video they’re seeing.

But I guess two responses to that that you might push back with, Sacha, one would be like, well, we’re just in a transitional period, right? This is new. This is so new to a lot of people seeing seeing this concept for first time that they just aren’t aware yet that this is possible and they’ll adjust over time. Another would be to say, look, yeah, maybe if I’m producing a cute image of, I don’t know, a rabbit or an image of someone breaking a bridge or something non-political, it’s easier to convince people that that’s real than it would be the case to, for example, change their political views. So mean, either or both of those responses, things you’d like to go with in response to that.

Sacha Altay: Yeah, I mean, my impression is that if a rando shares a video of Macron doing something crazy, people are not going to believe it. They are going to wait for like France Info and like the real media to cover it. Because if I don’t know, Macron is saying, we are starting a new war with this country. People are not going to believe it, even if it’s very high quality, because they know if it happens, all the media are going to cover it. So I think in very in this high case of like the politician saying something absolutely crazy, people are going to be vigilant and are going to wait for the mainstream media to buy it.

I think many of the AI slop that we see a lot on Facebook, but also on TikTok, are humorous ones. I think there is some part of the boomers, but not just the boomers, who want to be entertained. And for entertainment, they don’t really care whether it’s true or not, whether it’s authentic or not. And you have extremely, you can create extremely cute images of like little animals doing cute stuff and you get what you wanted. You have like this super stimuli, super cute, super entertaining, super engaging. You have what you wanted. Like, and whether it’s authentic or not, I do care. I don’t understand how people don’t, but at the same time it’s like, yeah, it’s brain candy. It’s a candy, brain candy that people get and I don’t see why it’s wrong.

And I just want to point out that we as elites, because we have always looked down on the content that the mass population consumes. Now we look down on like short video formats on TikTok, but we have always looked down on their entertainment practices, et cetera, saying that it makes them stupid, et cetera. And so I think we should be careful about that. Careful about saying that kids are stupid because they are on TikTok and are watching short format video or whatever. I think we should be careful. And I think we are falling a bit into that with the AI slop, but the TikTok AI slop are very different from the Facebook AI slop. The TikTok AI slop are very weird and absurd. And I think they work because they are extremely weird and absurd. You know, they’re something weird about them and people are playing with it. They are playing with the fact that it’s AI and that you can do extremely weird stuff, but it’s very different from the AI slop on Facebook that works, I think, among older populations.

Henry Shevlin: Since we’re discussing TikTok, just a quick point that’s been lurking in the back of a lurking worry I’ve had is it seems to me most of the research focuses on adults. And yet a lot of the worries about both social media misinformation and generative AI misinformation concerns teenagers and young people. And I’m curious, A, whether there’s how much specifically targeted research there is looking at that group. And B, I think there probably are some good prior reasons for worrying about that group more than others, just because teenage years—firstly, our political beliefs are less likely to be stabilised at that point. And secondly, it is obviously an important window for the formation of political identities in the first place. So even if the worries about social media and generative AI misinformation are overblown for adults, could there be more to worry about there in the case of teenagers?

Sacha Altay: No, that’s very possible. That’s a point that has at least has been made for social media and mental health that very few studies have looked at adolescents or young adolescents and that’s probably the group that’s like could be the most sensible to these effects. And so that’s a totally fair point. Regarding generative AI, I also think we should acknowledge that they are also probably much better at using the technology and recognizing it, like whether it is ChatGPT, DALL-E or like all the AI technology, I think they are much better.

And that’s why the AI slop I see on TikTok are like very meta, like they are second degree, third degree, like very meta. Whereas on Facebook, they are just like first degree, like, look, I did this amazing thing. Oh look, this cute baby. So I think very different. So to be honest, I’m not so worried about teens and generative AI on TikTok. Regarding mental health, I don’t know and we need more data, but it’s a very fair point.

Dan Williams: Just on this point about quality, so we’ve been talking about deepfakes, but there’s this other aspect of generative AI, which is just producing kind of tailored text-based content. And there has been this flurry of empirical research, so I’m thinking of like Tom Costello’s work on chatbots and conspiracy theories and so on, work by people like Ben Tappin showing that LLMs can be pretty persuasive with the content that they produce, partly because they’re just very good at recruiting evidence and persuasive rational arguments that is tailored to people’s specific pre-existing beliefs and informational situation. What’s your feeling about the impact of generative AI there? Because presumably there, it’s a very different conversation about deepfakes. And it does seem to me at least that generative AI, you might argue, is going to disproportionately benefit people with sort of bad, misinformed views, because that’s often where you’re lacking kind of human capital, right? You don’t have access on tap to the sophisticated intellectual skills of the intelligentsia when it comes to a lot of this kind of lowbrow misinformation. So they can now access, you know, generative AI, at least if it’s not subject to various sorts of like safety and ethical requirements, and that might happen down the line, isn’t there a real risk there that that’s going to kind of asymmetrically benefit people pushing out misinformed conspiratorial narratives?

Sacha Altay: So it’s good you mentioned these studies because they find super large effect sizes on important topics like politics. But all the authors acknowledge that these effects are estimated in experimental settings and it’s unclear how this would translate outside of experimental settings where LLMs are not going to be prompted to convince participants or users of believing something.

So first, they are not going to be prompted to do that. Second, they are not going to be paid to pay attention and use the LLM in that way. And so that’s why also, you know, Ben Tappin has this piece on like for mass persuasion, it matters more like attention. Are people actually going to do that? Are people actually going to be exposed to that rather than how persuasive it is? And that’s why I’m not so worried.

And it’s important, I think you mentioned the symmetry or asymmetry because I don’t see any good reason why bad actors would be more successful in using generative AI to mislead than good actors using generative AI to inform and make society better or citizens more informed, et cetera. I think in general, good actors have more money, have more trust. Like in France, if the French government releases an AI or whatever to inform people, it’s going to be more successful than if it’s the Russian government. And so in many ways, I think good actors have the advantage, but they need to take it seriously. They need to act and they need to proactively use these tools for democracy and for the better. They should not wait, I think, for the bad actors to attack and them to defend. They should already be using them in the best possible ways to improve society.

Dan Williams: Yeah, my thought concerning asymmetry was just take something like Holocaust denial, right? I think to a first approximation, everyone who believes in Holocaust denial is like stupid for the most part. And if you give them access to highly intelligent generative AI tools, well, they’re gonna be able to use the kind of on-tap intelligence to rationalize that false perspective. Whereas when it comes to the truth, namely that the Holocaust actually happened, we can use generative AI maybe to improve the persuasiveness of the arguments that we’re going to generate, but we’ve already got extremely persuasive evidence and arguments, right? Because that’s where all of the intellectual research and so on exists.

In any case, again, I’m conscious of time. Could we end with this point about personalization? So I still meet people who think that Brexit was due to Cambridge Analytica and micro-targeting and things like this. I think it’s a very kind of common belief people have, which is that once you start targeting personalizing messages, you can have like really huge impact on what people believe. And one of the consequences of AI, very broadly construed, is that they’re gonna greatly enhance the personalization of persuasive messages. So what’s your take on that?

Sacha Altay: Maybe the best evidence is actually the papers by Ben Tappin, Tom Costello and stuff who have actually measured what matters more. Is it whether the arguments generated by the LLMs are targeted to the users based on their political identity, etc., or whether they present more facts and the quality of the facts, etc. And in general, what they find is that what matters is facts. So the more you provide people with facts and good arguments, the more they change their mind. And personalization matters very little.

And in political science, there’s a whole literature showing mostly the same thing, that like, of course you need some targeting, like you need to target based on the language or like some basic level of targeting is needed, but like micro-targeting based on like, yeah, political preferences, values, et cetera, broadly ineffective basically, especially compared to the most convincing arguments you can make.

I think also there is a whole literature in like communication showing that people highly dislike targeted messages when they are very targeted, when they feel like it’s very targeted at them, people recognize it and they dislike it. Yeah, the Cambridge Analytica thing is just a scam basically. I still don’t know why people believe it that much. It’s just a company. They are selling influence. They said they influence major elections and all of a sudden people are like, oh yeah, of course I understand why they do that. People have priors about other people being gullible and being swayed by social media. So when a company said that they sway people on social media, people are receptive to it. They’re not being gullible. It’s just on their priors, et cetera. But yeah, no, there is very little evidence that Cambridge Analytica affected the Brexit or the 2016 US presidential election. And it’s better to present people with good arguments and facts rather than to micro-target them.

Henry Shevlin: If I can squeeze just another angle into the personalization discussion, something you talk about in the paper is relational factors, which is sort of related to personalization, but a bit distinct. And I’m curious about whether you think AI could play a role there. We’ve talked on the show previously about social AI and the idea that young people in particular might be forming deeper and more profound relationships with AI systems or AI friends, companions, lovers, which then potentially could be leveraged for changing their views.

And it seems to me just intuitively that these kind of relations, whether they’re sort of direct relations or more like parasocial relations, can be really influential if we think about, for example, something like Logan Paul’s Prime Energy Drinks. You know, this was an influencer who promoted his own brand of energy drinks that then became a massive sensation, hundreds of millions of dollars, if not billions of dollars in sales over a very short period of time. So it seems like these relationships can be powerful. Is that not a worry that AI could leverage them?

Sacha Altay: And to be honest, I’ve been, it’s very hard, it’s a very hard question. I’m being asked that all the time. And I think the best counter-argument I have at the moment is just, there is very little evidence that people change their mind according to their life partner. Like the people they trust the most, they sleep with, et cetera. There is very little change of mind. And when there is, it’s hard to know whether it’s because the incentives are getting more aligned. Like, you know, they get married, so they are sharing their money, they are buying a house together, they live in the same place, etc. So of course when the incentives are getting closer, you could imagine their beliefs, etc. are getting closer. But basically attitude change is very small with your life partner.

And I imagine that if my wife, who I trust a lot, I love, etc. tells me, GMOs are bad, nuclear energy is bad, etc. Why would she convince me? Like I trust her a lot on many things, but I’m not like completely blind to her. And so how would ChatGPT beat my wife at this, like, I don’t see it, I don’t see it. But to be honest, it’s just my opinion, let’s see how it goes, but I don’t find it very convincing.

Dan Williams: I can confirm that my girlfriend would very much like to influence my political attitudes, but is not having much success as of yet. Okay, one thing we didn’t do actually is you’ve given us your kind of analysis and your belief, Sacha, about the impact or lack of impact of generative AI. But we should mention there were all of these sort of alarmist forecasts about the impact of generative AI and deepfakes on the kind of 2024 election cycle.

And one of the things that you do in your paper is you don’t just go through each individual worry, but you actually kind of survey what the empirical research that we have says. So briefly, what does the research that we have actually say about the impact of generative AI on that election cycle?

Sacha Altay: I mean, to be honest, it’s not like a systematic review, like it’s not super reliable. I just went over and looked at what happened in these elections. And basically, in most countries, the consensus is that there have been some problems with elections, but that it’s old problems with elections, such as politicians lying, trying to gain, to change, like basically politicians doing bad stuff. And generative AI has been used a lot to illustrate what politicians want to say. Often they want to say that they are strong and that their opponent is weak or stupid. So they have been using generative AI to do that in the US, in Argentina, in many countries. They have used generative AI a lot to do some kind of like soft propaganda, portraying themselves and their group as good and the others as bad.

In some countries, apparently, generative AI has been used to do some good stuff, like in India, where we have like many languages and where translation is often a problem and takes time. And apparently, generative AI has been used a lot to translate some political campaigns into all the languages and dialects that exist in India. So I think it’s very varied and not as catastrophic, let’s say, as the alarmist tech suggests. But I think it’s just suggestive evidence. And of course, it’s just the beginning of generative AI. So we should see how generative AI will be used in the future, in future elections. But we should not forget that it can be used to do good stuff. Like it’s not necessarily being used to do bad stuff. You can use it to translate to, and even to illustrate, you can use it to do like faithful imitation, illustration. You don’t need to like portray yourself as super strong and the opponent as bad. You can do, I don’t know, some good or artistic stuff.

Dan Williams: Yeah, we didn’t really talk about the positive side of generative AI very much in this conversation. But my view is, at the moment at least, the kind of boring truth about large language models is that they’re basically just improving people’s access to evidence-based kind of factual information. And I think if you compare the kind of like one-shot answer you get from ChatGPT or Claude or Gemini on any political issue to what you get from the average voter or pundit or politician, it’s just of much higher quality. But I think that truth doesn’t really get the attention that it deserves because it’s sort of boring for the most part. It doesn’t fit into these kind of threat narratives. And it’s kind of counterintuitive because like, why would it be that these, you know, profit-seeking companies that everyone despises have just had a really beneficial consequence on the information environment? But that is in fact, what I think the case is.

Sacha Altay: So you’re totally right because another concern I haven’t mentioned is just hallucinations, like individual users using LLM on their own and being misled by an LLM because they confidently say stuff that is false. But as you say, I think it depends compared to what? How often do they hallucinate and how correct are they compared to alternative sources of information like other human beings, social media, TV?

And I think they would do pretty well actually compared to most of these other sources. And so that’s why I’m not so worried. I think the confidence thing is a bit annoying, but I think most people who use AI regularly kind of know that, yeah, sometimes they completely hallucinate and they go completely awry, but we know it. And I think most people who use it often know it. And that’s why I’m not so worried. But again, it would be better if they did not hallucinate and were perfect, but it’s setting the bar a bit high.

Dan Williams: Okay. Okay, fantastic. Well, thank you, Sacha. We’re going to have to bring you back on at some point because I feel like we’ve just barely scratched the surface with many of these issues. Was there anything that we didn’t ask you that you wished we had asked you?

Sacha Altay: No. I mean, as you said, many things to talk about.

Dan Williams: Okay, fantastic. Well, thanks, Sacha, and we’ll see everyone next time

If you look at humanity, both today and throughout history, you can’t help but notice that people believe a lot of things that seem stupid and irrational. Pick your favourite example: conspiracy theories, religion, prejudice, ideology, pseudoscience, ancestor myths, people who hold different political opinions from your own, and so on.

This observation provokes a central question for the social sciences. Why do broadly rational people, people who often seem intelligent and competent in most aspects of their lives, sometimes believe highly irrational things?

One classic answer is that people are not disinterested truth seekers. In some contexts, our practical interests conflict with the aim of acquiring accurate, evidence-based beliefs. For example, we might want to believe things that make us feel good, that impose a satisfying order and certainty on a complex world, that help us persuade others that we’re noble and impressive, or that win us status and approval from our friends and allies.

Famously, when our goals come into conflict with the pursuit of truth in this way, the truth often loses out. We lie to ourselves, bury our heads in the sand, and engage in elaborate mental gymnastics. Less colloquially, we engage in what psychologists call “motivated cognition”: we—or our minds, at least—direct cognitive processes toward favoured conclusions, not true ones. For example, we instinctively seek out evidence that confirms those conclusions (confirmation bias), shield ourselves from evidence against them (motivated ignorance), insist on higher standards for arguments we dislike than for those we like (biased evaluation), and remember and forget information in convenient patterns (selective forgetting).

Throughout most of history, scholars had little doubt that this tendency was a central and destructive feature of the human condition.

For Adam Smith, for example, it “is the fatal weakness of mankind” and “the source of half the disorders of human life.” For Socrates in the Cratylus, “the worst of all deceptions is self-deception.” And of course, thinkers such as Freud and Nietzsche placed motivated cognition at the centre of their understanding of human psychology.

Against Motivated Cognition

This consensus continued from the emergence of scientific psychology until relatively recently. In the last decade or so, however, some researchers have become increasingly sceptical that motivated cognition is a significant force in human affairs. There are many reasons for this, including reinterpretations of experimental findings, failures to replicate certain findings, and a growing body of evidence that people are broadly rational in how they process information, even in domains like politics.

I’m not very convinced by these sources of scepticism. I think they often rest on naïve assumptions about how to interpret psychological findings and how to understand motivated cognition. When properly understood, motivated cognition is consistent with the finding that people update their beliefs when presented with corrective information.

I also think that, as with human cognition more broadly, most widespread and consequential forms of motivated cognition are distributed and socially scaffolded. They are a “team project” involving complex systems of social norms, incentives, and coordination that function to promote and protect favoured narratives and belief systems. So, if you want to understand how we lie to ourselves, you must move beyond the lone thinker in decontextualised psych experiments and focus on how humans co-construct social worlds optimised for scaffolding self-deception.

There is much more to say about all of this, obviously. But here, I want to focus on a different, more “philosophical” source of scepticism about motivated cognition, one which draws attention to what the philosopher Jeffrey Friedman calls “epistemic complexity”.

Epistemic complexity

“Epistemic complexity” is a bit of jargon for the simple idea that it’s often really hard to figure out what’s true.

Partly, this is because reality itself is often complex, but it’s also due to the highly fallible ways in which we access that reality. We rarely have “direct”, perceptual access to the facts we form beliefs about, especially in domains like politics and religion. Our access is mediated by other people and institutions—priests, teachers, writers, journalists, pundits, scientists, social media feeds, etc.—and by our pre-existing beliefs (“priors”), which, given the world's scale and complexity, typically involve highly selective, low-resolution compressions of reality. Of course, these representations were also primarily acquired from others who are in exactly the same situation.

This is what Walter Lippmann meant when he observed that the modern world is “out of reach, out of sight, and out of mind”, and that public opinion “deals with indirect, unseen, and puzzling facts, and there is nothing obvious about them.”

To make this concrete, consider your beliefs about climate change. Maybe you think it’s our most pressing political problem, an urgent crisis and existential risk, or maybe you think the whole thing is an overblown, leftist moral panic. But whatever you believe, take a moment to reflect on where your beliefs came from.

Reality didn’t just imprint itself directly on your brain, whatever that would mean. You learned about climate change in the same way that you learn about almost everything else: through a highly path-dependent process in which, at every stage of encountering new information (testimony, news reports, articles, education, political commentary, etc.), you filtered it through your priors about the world and about which sources were trustworthy.

Through this process, you arrived at your current opinions, which inevitably take the form of low-resolution compressions of an extremely complex geophysical and political reality into a manageable, understandable form. Indeed, unless you are someone with significant expertise in this area, your “opinions” probably involve little more than socially-learned slogans and soundbites. (To test yourself, open a blank document and write out your current understanding of the topic exclusively from memory.)

It doesn’t take a philosophy PhD to appreciate that this process is highly fallible. Once you realise that the pictures inside people’s heads aren’t simple reflections of reality but the output of complex and fragile processes of interpretation and social learning, you should recognise that there are countless reasons why those pictures might distort or misrepresent that reality.

And yet, most of us don’t intuitively think this way. When we compare our beliefs against the facts, we always find a comforting 1:1 correspondence. Unless we force ourselves to reflect, there doesn’t seem to be a highly fallible process mediating between reality and our representations of it. Reality just is whatever we represent it to be.

The truth often seems obvious, self-evident, so much so that we are frequently baffled when people don’t share our understanding of the truth. The idea that rational people could have encountered the same reality and come away with different opinions doesn’t even register as a serious possibility in many cases. In the language of modern psychology, we are instinctive “naïve realists”. As Karl Popper characterised this intuition, we believe that the truth is “manifest”. If others don’t see the truth, they must, therefore, be deeply irrational, if not outright psychotic.

Given this, epistemic complexity is not merely a feature of our situation that we must grapple with. It is a feature that most people don’t instinctively appreciate, let alone reflect on. That is, it seems much easier to become “informed”—to figure out what’s true—than it really is.

Back to Motivated Cognition

This is where these reflections on epistemic complexity become relevant to questions about motivated cognition.

Historically, scholars have invoked motivated cognition to explain why people hold mistaken beliefs that appear highly irrational. If people confront epistemic complexity, this appearance of irrationality may simply be an illusion produced by naïve realism. That is, once we appreciate that the truth is not self-evident and that it’s extremely challenging to acquire knowledge, we should realise that there is nothing deeply puzzling about why people hold mistaken beliefs. Even perfectly rational individuals will form such beliefs if the challenges of forming accurate ones are sufficiently severe. Perhaps, through no fault of their own, they have simply been exposed to misleading evidence or unreliable sources.

If so, the motivation for positing motivated cognition evaporates. There is no irrationality to explain.

Although this move takes various forms, I think one can find versions of it in the writings of many recent scholars, even when it is not stated explicitly, including Jeffrey Friedman, Neil Levy, C. Thi Nguyen, and Cailin O’Connor and James Owen Weatherall. The core idea is that theorists have traditionally been too quick to jump from observing false beliefs to inferring motivated irrationality. Once we recognise epistemic complexity, we can see that there are countless ways in which individually rational thinkers can acquire false beliefs.

In most cases, these theorists advance alternative explanations that focus on features of the social environment, including how social-informational networks of trust and testimony are corrupted by malicious actors. Hence, this move typically goes hand in hand with the idea that to understand why people hold mistaken beliefs, we should turn our attention away from individual rational failings and toward “structural” and “systemic” pathologies in our society. (Friedman is an exception here, inasmuch as he seems to think that epistemic complexity is so severe that theorists shouldn’t even make judgements about which beliefs are true or false in the first place.)

Motivated Cognition and Epistemic Complexity

It’s an interesting and insightful line of reasoning, but I think it draws the wrong lesson from a recognition of epistemic complexity. Although such complexity opens the possibility that false beliefs can result from rational belief formation, its existence should actually increase our confidence in the likely impact of motivated cognition. This is because epistemic complexity exacerbates motivated cognition, making it easier for us to become convinced of desired conclusions.

In plain terms: The more challenging it is to figure out what’s true, the easier people will find it to lie to themselves.

To see why, think about the factors that determine whether people will engage in motivated cognition. It’s tempting to think that the only relevant variable is the strength of motivations that conflict with the pursuit of truth, such that the stronger those motivations, the greater the propensity to engage in motivated cognition.

However, a moment’s reflection suggests this can’t be the whole story. There are severe limits on what we can convince ourselves of, and these limits are largely independent of the strength of our motives. As Ziva Kunda puts it, “People do not seem to be at liberty to conclude whatever they want to conclude merely because they want to.” One might add: and no matter how much they want to. That is, there is no amount of money (or status, sex, etc.) that could induce me to believe that 2+2=5 or that the moon is made of cheese. These beliefs simply don’t fall within my cognitive grasp.

The reason is simple: For motivated cognition to be possible, we must be capable of providing some justification of the relevant belief. Elsewhere, I have called this a “rationalisation constraint”. But in some cases, we can satisfy it not by explicitly constructing or seeking post hoc rationalisations, but simply by insulating ourselves from disconfirming evidence. (This is captured by the “burying one’s head in the sand” metaphor).

Whatever we call it, however, the point is the same: our ability to become convinced of desired conclusions depends on our ability to feel that they are in some sense justified. That’s why the psychological acrobatics associated with motivated cognition—confirmation bias, biased evaluation, selective forgetting, etc.—are necessary in the first place.

For this reason, the extent to which motivated cognition biases belief depends not only on incentives but also on how easily individuals can satisfy this constraint. To be clear, this isn’t an original point; it’s one of the oldest observations about motivated cognition. The observation that I want to make here, however, is simply that when it comes to justifying desired beliefs, epistemic complexity is a help, not a hindrance. That is, as it becomes increasingly difficult to determine what’s true, it becomes correspondingly easier to convince ourselves of desirable untruths.

This suggests that many people are drawing the wrong lesson from epistemic complexity. Although such complexity implies that even disinterested, rational truth seekers could acquire inaccurate beliefs, its existence should increase our confidence that people are not behaving as disinterested truth seekers.

Of course, it is still ultimately an empirical question to what extent motivated cognition is operative in specific cases. There may be other reasons to think it is less prevalent than many have traditionally assumed. The point is just that the fact of epistemic complexity is not one of them.

So what?

Why does any of this matter? There are potentially many reasons, I think, but I’ll end with two.

First, it’s plausible that elites often benefit when target audiences engage in motivated cognition. So, politicians who spread self-serving lies benefit when their supporters prioritise political tribalism over accuracy. For example, they will be more likely to believe that an election was stolen from their side if they’re motivated to embrace and signal tribal beliefs. This means that many elites have an incentive to do whatever they can to increase a domain’s epistemic complexity—for example, by manufacturing uncertainty, flooding the zone with shit, recruiting congenial “experts”, and so on.

This is a familiar lesson from research on propaganda in some ways, of course, but reflecting on the interactions between motivated cognition and epistemic complexity casts it in a new light.

Second, many studies of the role of motivated cognition in belief formation provide participants with corrective evidence and measure the extent to which they update their beliefs. If they update in a rational direction, this is taken as evidence against the importance of motivated cognition.

One way to understand such experiments is that they artificially and temporarily reduce epistemic complexity. By presenting strong evidence against desired conclusions, they momentarily weaken people’s ability to subjectively justify those conclusions. To the extent that the real-world context in which people think involves much higher levels of epistemic complexity—for example, greater choice over which media and political sources to consult, heightened exposure to conflicting viewpoints and arguments, and greater contact with like-minded friends and colleagues—this suggests that such experiments might be limited in what they can tell us about the real-world significance of motivated cognition.

Chapters

• 00:00 Understanding AGI: A Controversial Concept

• 02:21 The Utility and Limitations of AGI

• 07:10 Defining AGI: Categories and Perspectives

• 12:01 Transformative AI vs. AGI: A Distinction

• 16:15 Generality in AI: Beyond Human Intelligence

• 22:13 Skepticism and Progress in AI Development

• 28:42 The Evolution of LLMs and Their Capabilities

• 30:49 Moravec’s Paradox and Its Implications

• 33:05 The Limits of AI in Creativity and Judgment

• 37:40 Skepticism Towards AGI and Human Intelligence

• 42:54 The Jagged Nature of AI Intelligence

• 47:32 Measuring AI Progress and Its Real-World Impact

• 56:39 Evaluating AI Progress and Benchmarks

• 01:02:22 The Rise of Claude Code and Its Implications

• 01:04:33 Transitioning to a Post-AGI World

• 01:15:15 Predictions for 2026: Capabilities, Economics, and Politics

Transcript

• Please note that this transcript is AI-created and may contain minor mistakes.

How Close Is AGI?

Dan Williams: Welcome back. It’s 2026, a new year, a big year for AI progress, an even bigger year, dare I say it, for this podcast. I’m Dan Williams. I’m back with Henry Shevlin. And today we’re going to be talking about one of the central, most consequential, most controversial concepts in all of AI discourse, which is AGI, artificial general intelligence.

So AGI is written into the mission statements of the leading AI companies. OpenAI, for example, states that their mission is to ensure that artificial general intelligence benefits all of humanity. We also constantly see references to AGI in the media, in science, in philosophy, and in discourse about the dangers, potentially catastrophic dangers, of advanced AI. And yet, there is famously very little consensus on how to even understand this concept, let alone measure our progress towards it.

Is it, for example, a system that achieves something called human level AI? Is it a system that can do any task or at least any intellectual task that a human being can do? Is it a system that performs extremely well on tests, on benchmarks? Or is it, as some people suggest, a deeply confused pseudoscientific concept? So for example, the influential cognitive scientist Alison Gopnik has said, there is no such thing as general intelligence, artificial or natural. Jan LeCun, one of the most famous AI researchers in the world, says this concept makes absolutely no sense.

But if that’s the case, what should we make of people making predictions about when we’re going to reach AGI, perhaps in the next few years? How do we make sense of rapid AI progress? What are we making progress towards? Moreover, what do we make of people, smart people, who claim we’ve already reached AGI, that we’re living through the post-AGI world?

So these are the topics that we’re going to be focusing on today. What is AGI? Is the concept coherent and useful? How do we measure progress towards AGI if we take this concept seriously? And what happens when or if we reach AGI? At the end, Henry and I are also going to be giving some predictions about how we expect AI to develop over the course of this year.

Okay, so to kick things off, Henry, AGI, how do you understand the concept? Are you a fan?

Henry Shevlin: I am a cautious fan of AGI as a concept. I think it’s an imperfect concept and can be very vague or defined in various ways. But at the same time, I think it serves as a useful reminder that we are heading towards an era, in my view, of genuinely transformative capabilities in AI systems. And so when we talk about AI revolutionizing science, AI revolutionizing medicine, AI revolutionizing the future of work, I think AGI is often a useful shorthand for talking about the point at which we start to see really massive changes in these domains.

That said, I do have some sympathy for the worry that this is not a particularly coherent concept. So I think we’ve seen commentary in the media recently saying, look, we don’t really understand what intelligence is, and therefore the very idea of AGI is ill-defined.

What I would say there is that I think we don’t need to understand exactly how human intelligence works in order to recognize when we’ve exceeded human capabilities in certain key ways. And in the same way, we don’t necessarily need to have a perfect biomechanical model of how birds fly in order to build planes that can fly faster than them. So I think even with some empirical questions or some conceptual or definitional disagreements about what intelligence is, what human intelligence is, it could still be the case that we’re well on our way to exceeding the capabilities of human intelligence across the board with AGI.

One thing to quickly flag though is AGI is kind of canonically or classically defined as systems that are equal to human level performance across all domains. I think tacitly this is often restricted to sort of economically and scientifically and cognitively relevant domains, right? So I think if we had systems that were sort of at human level or above in pretty much every cognitive task, but they couldn’t smell or had limited ability to do certain kinds of fine-grained motor tasks, perhaps, I think that wouldn’t disqualify us from characterizing those systems as AGI. If they’re doing better science than a human, if they’re winning mathematics prizes, if they’re Nobel’s, if they’re doing 99% of current jobs in the economy, it’s not going to be a deal breaker whether or not they can tell Sauvignon Blancs from a Chardonnay with a sniff.

Dan Williams: Yeah, although on that point, I think there’s a question here, which is, should we expect a system that can out-compete human beings when it comes to what are thought of as purely cognitive tasks, if it doesn’t have the kinds of competencies that go into, for example, folding laundry, making toast, et cetera. So the idea that you can draw a nice distinction between purely intellectual tasks of the sort that you can perform on a computer, and let’s say what I thought of as sort of non-intellectual tasks, of sensory motor tasks. I think that’s a kind of interesting question in and of itself.

Doing a bit of reading around AGI for this episode, it seems like a lot of the definitions about what AGI is splinter into sort of three different categories. And I’ll be interested to hear what you think about this way of taxonomizing area.

So some people seem to understand AGI basically as a kind of placeholder for whatever AI happens to have really transformative consequences. So it’s like, AGI is just a term for transformative AI, whatever form that transformative AI actually takes. Other people seem to understand it with this concept of human level AI or something similar, where they’re sort of using human intelligence as the thing relative to which we should understand the concept of AGI. And that I think for reasons we can probably get into, I can kind of understand what they’re getting out there, but I think there are all sorts of reasons to be skeptical about that concept. And then there’s a third category of attempts at understanding this concept where you’re just understanding it in terms of kind of abstract capabilities, right? And it might in fact be the case that human beings exhibit or instantiate these capabilities. But the idea is you can specify what these capabilities are independent of thinking about the specific form that they take in human beings. So things like the flexibility and generality of problem solving ability or capacities for continual learning and self-directed learning and autonomy and so on. So it’s like transformative AI understood in terms of impacts, you’ve got kind of human level AI where it’s a system which in some ways has capabilities that are like the sort of ones that human beings have, or you’ve got just a kind of pure capabilities understanding.

Does that correspond with how you’re thinking of this area? Would you add any other categories to that?

Henry Shevlin: Yeah, I think that’s really helpful. I guess a fourth category you might add, it’s a bit of a misnomer to call this category AGI. But I think in practice and a lot of discourse, sometimes people use AGI to refer to something like the singularity or some kind of recursive process of intelligence self-improvement. At which point, AGI functions basically as the same as the idea of artificial super intelligence. I think that’s probably not a maximally helpful way of thinking about AGI. I think it is helpful to distinguish between AGI and sort of the singularity or recursive intelligence explosions. But in practice, that’s what some people mean, I think, they talk about AGI.

Dan Williams: Yeah, just to add a footnote to that. So this idea of an intelligence explosion, roughly speaking, the idea is you’re going to get AI systems that once they can substantially contribute to the process of AI R&D and improving AI systems, you’re going to get this rapid process of recursive self-improvement where every AI system is sort of iteratively involved in building better and better AI systems. We should actually, I think, do a whole separate episode on the intelligence explosion. Because I think reading around about AI, so much of what people are thinking about the future seems to depend on their assessment of like the plausibility of that intelligence explosion concept.

But yeah, so I think you might though think of that as part of that first category of sort of defining AI by its impact in a sense. So AGI by its impact. So there AGI would be, you know, whatever triggers this hypothetical intelligence explosion.

I mean, I think that in addition to what you said, a general problem with defining AGI in terms of the impact of AI, where you’re sort of neutral on what kinds of capabilities might produce that impact, it’s not really forward looking, that kind of definition, right? It’s in a sense, almost by its very nature, going to be backward looking. And it’s not really clear then what we should be searching for or how you would go about measuring AGI solely by looking at the capabilities of the AI systems themselves. So even though I think there is a place for this idea that we need to be thinking seriously about what a world will look like where you’ve got radically transformative AI, merely having this placeholder, to me at least, doesn’t seem that useful as a way of understanding this concept of AGI.

Henry Shevlin: Yeah, I agree. And I think the idea of transformative AI is a useful concept in itself, but I do think it’s worth distinguishing from the kind of more cognitive scientific concept of AGI for a couple of reasons.

The first is that I think you can achieve transformative AGI, sorry, transformative AI, even with quite narrow systems. So there’s this really interesting idea that was very, very central to a lot of AI discourse in the late 2010s called comprehensive AI services. So this is an idea developed by Eric Drexler who said, look, maybe it would be a good idea for safety reasons if rather than trying to build one AI to rule them all, we focus on more narrow domain expert AI systems. So you’ve got an amazing AI scientist, an amazing AI financial analyst, you’ve got an amazing AI writer, but they’re not joined up. They don’t talk to each other at least directly.

And that could be better from a safety perspective, but also pretty much just as useful as AGI. So this is often framed as sort of a choice between two different directions that the future of AI research could go. Part of the problem there is I think LLMs kind of fall between the cracks of AGI and CAIS, as it’s called, Comprehensive AI Services. But insofar as they are a sort of unified system in some sense in terms of their generality, they can do lots and lots of different tasks and they’re not narrow systems. But at the same time, they’re not unified in the sense of being a single psychological agent with memory carried across different instances, capable of coordinating thousands or hundreds of thousands or millions of different conversations towards a single goal. And of course, a lot of the power that LLMs have is their ability increasingly to use various tools rather than sort of having those tools integrated into the systems themselves.

So I think there’s a world in which AI turns out to be transformative that ends up looking a lot more like Eric Drexler’s world. So this is a world without AGI in the sense of, you know, one system to rule them all. Instead, lots of powerful specialized systems, but that still utterly transforms our society and economy. So that’s one reason I think the transformative definition is maybe worth separating out.

Another reason to separate out transformative AI from AGI is something that’s been a big issue in the last year, which is adoption. We could have amazing AI systems or increasingly powerful AI systems, but due to economic or structural factors, they don’t end up at least straight away having the kind of transformative impact that people I think sometimes slightly naively assumed would just happen straight away as soon as you get AGI. So again, you might have, so it’s a sort of a double dissociation. You might have transformative AI that still falls short of AGI because it’s something a bit more like CAIS. Or you might have genuine AGI, but it’s not yet or not immediately transformative because of structural, legal, economic obstacles, things like adoption, to prevent it having the full impact.

Dan Williams: Yeah, and I think that idea that you can’t leap straight from the capabilities of the system to its real world impact is a very important idea in thinking about AI in general. And in fact, we touched on this in our first episode where we looked at the AI as normal technology perspective from Arvind Narayanan and Sayesh Kapoor, where they make a really big deal of this idea that diffusion takes time. There are lots of bottlenecks. There’s going to be lots of risk aversion, need to have all of these other complementary innovations within society in order to actually integrate AI capabilities. I think that’s really important.

Maybe to sort of take a step back. So as I understand where the concept of AGI sort of first comes from, when we separate it from these questions of real world impact and just focus on the capabilities of a system, is if you look at the history of AI, we have lots of very impressive systems, often superhuman, along relatively narrow dimensions, but that could only do some things. So a chess playing system that will destroy the world’s best chess player, but it can’t really do anything else. And even if you just change the rules of chess very slightly, the systems are so brittle that suddenly they’ll lose all of their capabilities.

And one thought was, well, that’s one kind of intelligence, a kind of narrow intelligence, which these AI systems that we were building possessed. But in principle, there could be a kind of intelligence where it’s incredibly flexible and open-ended in terms of the kinds of tasks, the kinds of goals that the system could achieve. And then I take it a question people are gonna have is, okay, why should we expect such a system is even possible?

And a thought many people have is, well, we have human beings, right? And human beings are a kind of existence proof for a certain kind of highly general, flexible, open-ended intelligence, in as much as human beings can become poets, scientists, engineers, dancers, we can play an open-ended set of possible games, and so on and so forth. So the idea is there’s gonna be a kind of conceptual contrast between narrow intelligence and general intelligence. And in a way of addressing skepticism about the possibility of general intelligence, people can always say, human beings have this kind of generality in terms of the sorts of things that we can do.

And I take it that’s partly why so much of the AGI discourse gets translated into human level AI discourse because human beings are supposed to be this kind of existence proof for the kind of intelligence that we’re thinking about. I’m really torn here because I think clearly on the one hand it is true that human beings have a kind of flexible open-ended intelligence that can be combined with an open-ended set of goals and we can perform a variety of different tasks. On the other hand I do really worry about this concept of human level AI, it feels a little bit incoherent to me, like we’re dealing with a kind of great chain of being where there’s this single quantity of intelligence and human beings were on a certain level and we just need to get to that level. That feels a bit confused and sort of dubious to me.

I also think, and actually maybe this is an area where we disagree, ultimately it’s not obvious to me that you’re going to be able to build systems that can do everything that human beings can do that work radically differently from human beings and are subject to a totally different kind of design process in terms of the learning mechanisms by which they arise. I think that idea is coherent, but I think this concept of AGI is basically saying we’re going to get systems that can do everything that human beings can do. They’ve got the kind of flexible, open-ended intelligence, but they’re not going to work anything like how human beings work.

I feel like that idea doesn’t get enough scrutiny in discourse about AI. What do you think?

Henry Shevlin: So loads of juicy threads there. Just a couple of quick historical notes. So the idea of generality as a feature of AI systems was really popularized by John McCarthy all the way back in the sort 60s and 70s, one of the founding figures of modern machine learning and AI. And then I think AGI or the concept of general intelligence as a central notion for frontier model development is sort of popularized and refined a bit by Shane Legge and Marcus Hutter in the early 2000s. So they give this famous definition of general intelligence as the ability to achieve goals across a wide range of environments.

And if we’re going to sort of do any useful sort of scientific analysis, I think, with the concepts in this vicinity, I think the idea of generality as a sort of continuous dimension is more useful and interesting than the concept of AGI per se. I think the AGI sounds like there’s a definite finish line for model development, which I think is probably unlikely for reasons maybe we’ll get onto, but spoiler alert, I think it has to do with the jagged frontier and the jagged nature of AI development. But on the other hand, the idea of generality seems like a really legitimate scientific category, right? Be able to measure, you know, obviously operationalizing these terms is always a bit tricky, but the idea that we can measure the ability of systems to perform well across different domains, that seems like something that is measurable and is meaningful. And I think that’s an area where we’ve seen astonishing progress in very, very recent history.

So back in, I think it was 2019, I wrote a paper with Karina Vould, Matt Crosby and Marta Helena called The Limits of Machine Intelligence, where we were comparing contemporary frontier AI systems somewhat negatively with capabilities, not just of humans, but of non-human animals. In that paper, we draw heavily on biology and just talk about the wide range of things that honeybees can do that birds can do, how they are not specialized intelligences and comparing them with things like AlphaGo or AlphaFold, which are, as you sort of suggested, really, really powerful systems, but operating in very, very narrow domains.

Now, since then, and somewhat, I think, to the surprise of me and others, large language models have shown that in some ways it is possible to build really quite robust systems, systems with a very high degree of generality across a lot of cognitive tasks. And I think that this has sort of dawned quite slowly. I think as recently as sort of just like the launch version of ChatGPT, which was running on 3.5, you still ran into a lot of the kind of familiar problems that you’d run into with sort of previous systems that you alluded to. You change the rules of chess slightly and you get sort of inelegant failures. And I think you could see that already with things like ChatGPT, the launch version would often make non sequiturs. It was easy to confuse. Fairly trivial to get it to hallucinate. And across all those metrics, these systems have been getting more and more reliable.

Early ChatGPT was terrible at mathematics, for example. Contemporary ChatGPT or contemporary LLMs in general can do fantastic mathematics. We’ve had admittedly specialized fine-tuned models, but still LLMs at core that are now winning International Math Olympiad goals. So I think maybe one way to push back against your idea that generality, or at least your hypothesis that maybe generality, high levels of generality, are only achievable in something like a human package. Well, I think the trend line suggests that we are moving rapidly towards more and more general systems in a distinctly unhuman-like package in the form of LLMs.

Dan Williams: Completely agree. And this is, I think, the kind of strongest argument for the alternative view. I mean, just to kind of reconstruct my somewhat garbled reasoning, my thought was something like, we talk about AGI, and often the existence proof that there’s such a thing as AGI is the fact that we’ve got human beings. And I think so much of the discourse about why AGI will be transformative is the idea that these systems will be able to do everything that human beings can do, maybe just in the cognitive, intellectual domains.

And my thought was, well, fair enough, but we’re not building systems that work anything like the human mind, anything like how the human mind works. So there’s a kind of assumption here, sort of bundled with this AGI concept in terms of the way that it gets used, which is we’re going to build systems or we can build systems, maybe we are on track to build systems that can do everything that human beings can do in a way that this concept of AGI sort of captures, but that work nothing like human beings. And I don’t think it’s obvious that that assumption is true. A priori, certainly being a physicalist, being a functionalist doesn’t commit you to the truth of that. So the question is, why should we believe it?

And I think a very good response is look at what’s happening in AI over the past few years. Maybe a kind of skepticism made sense in 2020. But now, just given the realities of how much AI progress there’s been, especially when it comes to the generality of these LLM-based systems, that skepticism is difficult to maintain. I think that’s fair. I definitely think that the progress that we’ve seen in AI and the fact that clearly a significant aspect of this progress is the generality of problem solving ability with these systems. I think that does put pressure on the kind of skepticism that I was raising.

I do wonder how much pressure. Like suppose someone just wants to say, okay, you’ve made a certain kind of progress. We can characterize that in terms of generality. But of course, the people who are really bullish on AI progress, they’re not just claiming that these systems are very competent and general as we find them today, they’re claiming that we’re going to have drop-in workers that can substitute for human labor across different areas of the economy. Why should we extrapolate from progress that we’ve seen over the past four years and think that that’s going to get us to the full suite of capabilities that we associate with human intelligence? We’re kind of skipping ahead here to get to questions about benchmarks and progress and so on. But I think it’s an interesting question. What are your thoughts about that?

Henry Shevlin: Yeah, I think you’ve characterized the debate really well. And I think it was a really plausible hypothesis, even a couple of years ago, that, you know, to use the meme, the phrase that has rapidly become a Twitter meme, you know, deep learning is hitting a wall, LLMs are going to hit a wall, that it was like a really viable empirical hypothesis that we’d find out that there’s only so far you can go with these very unhuman-like architectures. Okay, maybe we find out that you can use them to generate high quality code and do basic composition and translation. But there is some sort of task set T where no matter how big we build the models, they’re just no good. Maybe that would be social cognition or causal reasoning or scientific reasoning.

And yet every candidate domain pretty much has fallen. So I think that doesn’t mean that we won’t find some candidate domains where it turns out these systems just, where just scaling these systems up won’t lead us to greater progress, but we haven’t found them yet.

I think probably the most interesting one that I’m watching at the moment is agency. Some of, I think, I’m not sure if we’ve discussed it before, but things like Anthropic’s experiment with Claudius getting Claude to run vending machines at Anthropic’s offices and failing abysmally any kind of like long term structured planning task that involves interacting with different human agents, some of which might have slightly malicious motives, you know, people trying to get discounts from the vending machine. It’s very funny. We can probably drop a link to the study in the blog. But that was an area where it looks like we really are struggling to build systems that can do something like sustain human agency. But even there, we’re seeing rapid progress. And it’s not clear to me that we’re immediately hitting any sort of brick walls.

So that said, it is entirely possible. And I’d also just emphasize again that I think this is very much an empirical question. I think, again, it was a really plausible hypothesis a few years ago to think that simply training on language alone wouldn’t be able to get you anything like cognition. I think there’s a natural vision of how cognition works where, in the human case, language sort of sits at the top of the pyramid. And then you’ve got layers underneath of things like sensorimotor cognition, motor skills, spatial reasoning, social reasoning and so forth. And language is just the capstone. And if you try and build that capstone without the supporting layers, sure, you might be able to do some clever stuff, but it’s never gonna give you real intelligence.

And I think the discovery that at least that doesn’t seem to be the case from what we’ve seen so far is from just a general cognitive scientific point of view, probably the most astonishing discovery in cog-sci in several decades, I think.

Dan Williams: Can I just quickly interrupt, Henry, because I really want to make sure that I’m understanding what you’re saying. So the last thing you said was you might have a model of kind of agency and intelligence where you need to get the sensorimotor stuff, the kind of embodiment, the being in the world. Is that a Heidegger phrase? I’ve no idea what he meant by that. But that kind of stuff, you need to get that basic sensorimotor stuff, lots of the stuff that we share with other animals, right, first before you can get these more kind of cerebral intellectual tasks like being amazing at software engineering and coding and mathematics and language and so on. And your thought was that was an interesting hypothesis. Actually, what we found with AI in the past few years is it’s not true. Actually, you can get all of that really kind of cerebral, highly intellectualized, those sorts of capabilities without that other stuff.

But couldn’t someone say, well, that sort of cuts both ways in a sense. So we’ve talked about this previously, but there’s this famous, you know, Moravec’s paradox, you know, things that we find easy are hard. Things that we find hard are relatively easy. And that what we found with AI progress over the past several years is, yeah, these systems have got really good with these kind of, we might think of them as evolutionarily recent capabilities that human beings have these very abstract cerebral intellectualized stuff to do with manipulating text and so on. But real, kind of, the significant challenge when it comes to intelligence isn’t that stuff, it’s that sort of basic sensorimotor coordination, these much lower level abilities that we share with other animals. And so far we haven’t seen much progress on those things. And therefore we shouldn’t actually be so bullish on the progress that we’ve seen with these AI systems over the past few years.

Henry Shevlin: Yeah, again, really interesting. I think Moravec’s paradox is looking a lot shakier than it used to. So for it, I mean, one example of something that was sometimes cited as sort of an instance of Moravec’s paradox was image recognition. Image recognition was famously incredibly, incredibly hard, correctly categorizing the kind of things that were in a presented image. And then around 2012, things like AlexNet was one of the early deep learning systems that started to radically tear away these benchmarks and start to dramatically improve on previous generations of performance. And I think it’s fair to say that image categorization is basically a solved problem now.

And I think in quite a few Moravec-type domains, we’ve seen very, very rapid progress. So another Moravec-ish domain is things like understanding conversational implicature or subtle things that people might mean. So conversational implicature, a technical philosophical term, but huge amounts of human language or huge amounts of human communication rely on things like theory of mind and shared context. So if I say, what do you think of that? Whether that is referring back to something I said five minutes ago, being able to figure out what I’m referring to, that’s a very Moravec-style skill that relies on a lot of contextual knowledge. But in these kind of domains, AI just does brilliantly nowadays. AI is very good at conversational pragmatics or conversational implicature, very good at image recognition.

So Moravec’s paradox is no longer, it’s no longer clear that it holds or it holds in a much more uneven and jagged way. It’s not the case that sort of everything that’s easy for a two year old is hard for AI and vice versa. So I think that’s one of the ways in which I would push back.

Regarding the broader question, sure you’ve built sophisticated language models. That doesn’t mean that these systems will then be able to do the fancy sensory motor stuff. I agree. I think that’s absolutely right. So it may not be the case that LLMs five years from now are any better. Well, I think they will be at least a little bit better at the kind of sensory motor stuff as we’re seeing from the increased integration of sort of LLMs into robotic architectures and so forth. But yeah, I think it’s definitely possible that we found a different way to build high-level intellectual capabilities that doesn’t translate to sensorimotor capabilities.

But the other thing I would flag here, and maybe this slightly undermines my own point from earlier on, is that contemporary LLMs are radically different beasts from LLMs three years ago. Contemporary LLMs interpret live video. They interact with the world via querying web results. They can access APIs. They can use tools. They are in a kind of dynamic relationship with the world, albeit one that’s a little bit different from ours. You can ask ChatGPT, is this bar open on a Friday? And it’ll say, yes, I think it is. And say, can you check that? And it’ll come back and say, I’m wrong. Sorry. Yes, they’ve just recently changed their opening hours. They’re now closed on a Friday. I think that is almost a form of sensory motor grounding, you know, in obviously a different package. But so I think contemporary LLMs are, they’re not just sort of these ossified monoliths trained on a bunch of text and then frozen in time forever. They are in some ways closer in, at least at a very abstract architectural level to the kind of dynamic, quasi-embodied systems that we are.

Dan Williams: Interesting. I’m not so sure that Moravec’s paradox has been challenged to the extent that you’re suggesting. I mean, we don’t, we don’t have robotics, right? It’s nowhere near as advanced as these LLM-based systems.

Henry Shevlin: Well, hold on, hold on. Just on that point, what do you think of driverless cars as a counter example here? Because driverless cars were another one of these things that where people often used the failures of driverless cars in the 2010s as an example of Moravec’s paradox in action. They said, these people, actually things like driverless cars are to be far harder than people realize because it involves this whole complex suite of sensory motor capabilities. But now, the safety record of Waymo in the Bay Area exceeds that of human drivers.

Dan Williams: Yeah, very, very good point. I do think though, some degree of goalposts shifting and realizing that certain things we thought would be very hard and much easier than we thought can kind of be legitimate in this context because our intuitions are not particularly reliable when it comes to tracking what really matters about intelligence.

So if you go back to the seventies and eighties, all of these people, even those who thought that embodiment was really central to intelligence, they would say things like, well, you’ll never get an AI system that can beat a human being at chess because that’s going to tap into all of this constellation of abilities, which are, you know, connected to our embodiment and so on. And then obviously we know what happened, but I think part of that is we’re just learning with every kind of development with these AI systems that there’s much more to intelligence than we thought. So yes, we do have self-driving cars, but we don’t have functional robotics of the sort that we can integrate into our lives, suggesting that self-driving cars as impressive as that kind of technology is, is not really a proxy for the kind of full suite of sensory motor abilities that we care about when it comes to animals’ interactions within the world.

I think we’ve also so far been thinking of Moravec’s paradox in terms of this contrast between the highly cerebral intellectual domains, kind of symbolic, often explicitly text-based and basic sensory motor control. But I think there are things like continual learning, right? The capacity of animals, very, very young children, a perfect example of this, to be constantly learning from their environments. And in a way, I think that’s one of these things which state of the art AI today hasn’t cracked. I mean, you’ve got this kind of pre-training phase where it’s next token prediction. Then you’ve got post-training where it’s various sort of reinforcement learning-based learning processes for the most part. But you don’t have kind of continual learning, updating of the model weights as they go through the world from their experience. And that’s not strictly speaking, just a sensorimotor thing. That’s also connected to our sort of higher abilities.

And then also things like, you know, creativity, judgment. We’ve got these words for these concepts. And I think our explicit understanding of them is quite weak. But I do think there’s something to the idea that, you know, ChatGPT, the amount of knowledge this system has is unimaginable relative to what an individual human being has. But individual human beings can do things in the cognitive domain, which is still much more impressive than what systems like ChatGPT or Gemini can do. And again, that’s sort of, it’s not, it’s a kind of competence, it’s a kind of ability, which is not purely sensory motor, but what I think is quite central to how animals in general go through the world, capacity for judgment, for creativity and so on, which again, these systems don’t seem to possess.

And one reason for that might be that they’re just these incredibly weird systems relative to human beings. Their training process is completely different. Their architecture is completely different. And they can do these things that are incredibly impressive, almost unimaginably impressive relative to a few years ago. But there’s a great quote actually from AI podcasting legend, Dwarkesh Patel, which is something like these systems are getting more and more impressive at the rate the short timelines people predict, but more and more useful at the rate the long timelines predict. The thought being, yes, what they can do in terms of our subjective sense of how impressive it is, is amazing. And they’re performing very, very well in terms of these benchmarks. But in terms of their real world utility, actually they’re not having the impact that many people think. And one reason for that might be that they lack many of these kind of amorphous, nebulous capabilities that human beings and indeed to some extent other animals have. Sorry, that was me. I’m not, that was very nebulous and sort of inchoate in terms of their thoughts there, but I’ll be interested to hear what you think.

Henry Shevlin: Well, can I ask, what are some examples of judgment or creativity involving tasks where you think contemporary models clearly fall short of human capabilities? And I’m not denying that there might be such cases, but I’m just curious if there are any ones you have in mind.

Dan Williams: Yeah. Well, for example, I mean, I’m a writer and a researcher. I don’t think AI systems as they exist today, or maybe I should actually, I should rephrase that as commercially released AI systems, because God knows what’s happening privately within these labs. I don’t think they could function as a researcher and as a sort of writer generating novel and interesting opinions, which is the kind of self image that I would like to have. I think they can write bloody well. And I think if you use them as an assistant, it can be incredibly helpful in terms of augmenting and enhancing your abilities. I don’t think we’re at the stage where a ChatGPT could function as a substitute for me, which in a way is strange because it has a knowledge base, which is sort of just so vast relative to my knowledge base or the knowledge base of any other kind of researcher.

So I would imagine if you took my abilities, limited as they are, but combined them with this kind of almost godlike knowledge base of the sort of these systems have, you would get really, really kind of impressive research outputs. But you just don’t see that when it comes to these state of the art AI systems. Am I missing something? Do you disagree?

Henry Shevlin: Well, I think one thing that’s worth mentioning is I think it might be a little misleading to compare you who are, you know, I think, I hope you won’t mind me saying an elite sort of knowledge worker, right, with in thinking about sort of your ability to do original composition, original essays, original analysis. Yeah, I think you still have an edge. But I think we’re well past the point where sort of the median undergraduate essay, I mean, the ChatGPT in its current form can produce far better essays than the median undergraduate essay. I think at this stage, in some domains, it can produce far better essays than the median grad student 5,000 word essay.

And so I think there’s a little bit of a tension there if you’re saying, humans in general have this special sauce that lets us do things that AI systems can’t, when in fact, already AI systems in the kind of domains you mentioned already do vastly better than the very large majority of humans within these tasks.

Dan Williams: Yeah, I think I’m more open to the possibility that they’re doing something very, very weird, incredibly impressive, that does seem to outcompete human beings across specific tasks, but they do in fact lack many properties and capabilities that human beings have such that they couldn’t substitute for them even when it comes to purely intellectual tasks. I do realize though, there is the possibility of a significant amount of copium, self-serving cope in terms of this. And there’s something unsatisfying about it as well, in as much as I think you’re right to, you’re really right to push back. And also I would say, I wouldn’t have predicted the progress of the sort that we’re seeing back in 2020. And I think I probably haven’t fully updated to the extent that a rational individual should have done given the kind of progress that we’ve seen.

But let’s just quickly sort of, let’s return to this, but I’m aware of the fact that we got derailed by a really interesting conversation there. And just take a sort of detour through, we’ve touched on this introductory stuff about kind of AGI. So, you know, how you might understand the concept in terms of transformative impact, in terms of human level AI, in terms of more abstract sort of functional specification of capabilities. Maybe we can just spend a little bit thinking about the skeptical arguments concerning this concept of AGI. So like people like Jan LeCun or Alison Gopnik that I mentioned at the beginning, just saying the concept makes no sense at all and there’s no such thing as general intelligence.

I take it, I mean, one argument you often find here is that human beings are supposed to be the existence proof for the AGI. Here is a, you know, complex information processing system that has the kind of set of capabilities that people that talk about AGI are interested in. But the thought goes, well, human intelligence is not general. The human brain is this integrated mosaic of very specialized abilities that correspond to the kinds of problems we confronted in our kind of evolutionary past.

Sometimes this is cashed out in terms of like massive modularity to get a bit nerdy in terms of the cognitive science debate. And I think people into that kind of perspective, they think there’s something problematic with the concept of AGI because it seems to assume that intelligence is this one generic problem solving ability when in fact human intelligence, which is supposed to be our only existence proof of AGI, doesn’t take that form. It’s this set of special purpose modules for different tasks, which might be nicely integrated in the case of the human brain, but don’t involve just general purpose sort of learning mechanisms. What’s your thought about that kind of critique or that kind of worry?

Henry Shevlin: Yeah, so I’m pretty sympathetic to massive modularity in the human case. I think if you are sympathetic to massive modularity in the human case, that just seems like one way of interpreting that is to say that AGI can operate across, or that general intelligence can operate across highly modular architectures. If what we’re thinking about when we’re thinking about general intelligence is something ultimately grounded in the ability to perform cognitive tasks, right? Does it matter whether that’s achieved purely via a relatively narrow bundle of cells all in your prefrontal cortex or using working memory, or if it’s a bunch of different sort of cognitive sub-modules working together.

So yeah, I think if you accept the massive modularity as a thesis in humans, then why not just say, okay, so maybe the way we get to artificial general intelligence is through a similarly massively modular system. And you can already see hints of this in the way that in the increasing tool use by AI systems.

And it may be that, and this sort of goes back to our discussions about CAIS versus AGI, that the first kind of true AGI systems, I’m skeptical we’ll ever have like a clear, we’ve built AGI moment. But maybe the first systems that sort of get most people would agree are AGI systems might similarly have a relatively modular architecture, maybe with sort of a central coordinator powered by an LLM coupled with a dedicated mathematics engine, coupled with dedicated deep reinforcement learning agents for doing various kinds of scientific work, coupled with, you know, maybe sensory motor systems embedded in drones for doing that kind of thing. I think that would still be AGI, at least in the sense that it’s sort of relevant and interesting.

Dan Williams: Yeah, that’s a very, I think that’s a very good response. I mean, is the worry then that these people have that actually, if you look at AI as it exists today, most of what’s powering it is very general purpose learning mechanisms that doesn’t really look like what you’ve got in the human case. So maybe we should be skeptical that you’re going to get to human-like capabilities via this architecture. But I think your point that actually there’s a lot more kind of modularity here than you might think if you just look at the base model precisely because of this interface with all of these mechanisms. I think that’s important.

I wonder if there’s another kind of thing in the background here, which is skepticism about the way in which AGI often gets talked about where it’s like, we’re gonna build AGI and it’s gonna have these almost superhuman capabilities across all of these different domains. And maybe some people think, well, if you look at human beings, existence proof for this concept of AGI, you don’t find anything like that. You find that we’re very good at some things, we’re not so good at other things. So maybe the thought would be once you’ve paid attention to how human intelligence and maybe more broadly kind of animal intelligence works, very kind of specialized, very modular, that should make you a bit more skeptical maybe about some of the claims about the capabilities of super intelligent AGI in the future. What do you think of that kind of argument?

Henry Shevlin: Yeah, I mean, I think it’s definitely worth stressing how sort of distributed our civilizational capabilities are across different humans, right? I think most humans are not fully general in their intelligence. Some people are great at mathematics, some people are great at coding, some people are great at languages. But we’re able to achieve remarkable things at the civilizational level or at the cultural level because of cooperation across different kinds of specialists within our massive population.

But again, I don’t see why a model like that couldn’t apply to AI systems. Maybe that’s across millions of different instances with different fine tuning to different tasks. So yeah, I think the lack of generality in individual humans is compensated for at the population level. And I don’t see why a similar kind of distributed architecture couldn’t apply to relatively near future AI systems in the kind of modular way I’ve been describing.

There’s an idea here worth bringing back that I touched on earlier on, which is the jagged nature of current AI systems. So for anyone who’s not familiar with this, roughly the idea is if you think about sort of a spider diagram or a radar chart, as it’s sometimes called, where you sort of think about different dimensions of intelligence and sort of map human performance on this, you know, we’ve got sort of spatial reasoning, mathematical reasoning. And let’s just say for the purposes of argument that humans are pretty well rounded across this domain.

AI systems, current AI systems are really, really superhuman already at some tasks, well below human performance on others, around human performance on some. I think this is a really striking observation and a really important observation for understanding trends in current AI. And also explains a lot about the point you made earlier about why these things are maybe less useful than you might have expected.

And, you know, I’ll happily say on the record here that I was far more optimistic about the near term economic impacts of things like ChatGPT, then turned out to be actually correct. If you’d asked me, well, I think I was saying back in November, 2022 on Twitter and places that this is going to revolutionize the economy in the next few years. I still think it is going to revolutionize the economy, but it’s been a lot slower than expected. And I think jaggedness is a big part of the reason, adoption is another.

But just to sort of go into this a little bit more detail, when we think about what an individual human job involves. It involves a huge range of tasks. It’s not one task for the most part. They’re bundled tasks. Current AI systems are really good at some and bad at others, which makes the idea of the drop-in agent-employee model currently non-viable because there are enough tasks within human workflow that AI is really bad at to mean that’s just not applicable.

So a couple of things you might say, how we’re to get around this problem. One is that just rely on these systems getting better and that jaggedness, if not smoothing out, then to sort of the sheer level of the abilities expanding sufficiently that, you know, even if AI systems are still vastly superhuman in some domains and only human level in others, they’ll be good enough across the board that they will be able to function as drop-in agents.

Another interesting idea though is that we will just redesign task flows. We will do some unbundling of tasks in roles such that we create sort of roles that AIs can be dropped in on quite safely. I think a nice useful analogy here, I was talking about this on Twitter not long ago, is if you look at mechanization in agriculture, right, it’s not the case that mechanization in agriculture proceeded through creating robot farmers. It involved instead changing task flows such that relatively simple machines could take over very sort of labor-intensive tasks from humans and changing the kind of things that the average human farmer does.

I think that might be a better model for thinking about at least near-term AI impacts on employment, where it’s a matter of redesigning task flows and value chains such that there are, we do create these niches where you can drop in these AI agents to take on huge important parts of the value chain without necessarily replacing humans one-for-one on the kind of jobs that humans currently have.

Dan Williams: Yeah, that’s really interesting. And I think it’s a very insightful observation. I mean, I would say though, when we’re thinking about what people do in their jobs, it’s not like, you know, there’s a set of tasks that are separate from each other and, you know, AI can do 40% of them or soon it’s going to be able to do 60% of them. The tasks are integrated with each other in an incredibly complex way, such that we might be able to delegate some of these individual tasks to an AI system. But if I think about my job as an academic at a university, it’s not like I can say, my job consists of 142 tasks and here they are. It’s a much more integrated kind of unified set of responsibilities and obligations.

So I think if we’re thinking about not just delegating some tasks to AI systems and adjusting how the workflow is structured and adjusting the structure of organizations, but thinking about radical forms of automation. At the moment, I think that we’re very far from that precisely because I don’t think even as impressive as these AI systems have been, they’re capable of that kind of really kind of long time horizon integrated, like multimodal task performance of the sort that most human beings perform.

And that actually gets us nicely onto something we’ve already touched on, but I think we should think about and talk about as a kind of separate topic, which is measuring progress in AI. So lots of this is framed in terms of progress towards AGI. But I guess you can just think of it in terms of the progress and the capabilities of these systems in general.

So I think there are kind of three overarching ways in which we do this, again, to draw another distinction between three different categories. There’s the kind of subjective, how impressive is this? It’s not completely without value, but I think it is very unreliable for various reasons. There are the sort of set of benchmarks, formal benchmarks that are used to evaluate model performance. And then there is actual kind of real world deployment. So something like what percentage, what fraction of work in the economy is done by automated AI systems or something like that.

If you’re thinking about those three categories, I take it that the quote that I paraphrased from Dwarkesh, where these models are getting more impressive at the rate that the short timelines predict and they’re kind of more useful at the rate that the long timelines people predict. That’s drawing a distinction between two different ways in which you can evaluate these systems. There’s the kind of how subjectively impressive do we find them? And maybe that’s also connected to benchmark performance, where as you’re saying, they’re just getting better and better at a seemingly just ever increasing set of tasks. It’s juxtaposing that with like real world utility. I think that’s complicated, as you said, by the fact that real world deployment is not a simple function of capability, is also going to be shaped by all sorts of other things.

But how are you thinking about this, about measuring AI progress, how to use that to forecast AI progress?

Henry Shevlin: Yeah, I think that’s a fantastic tripartite way of splitting it up. So just a couple of quick comments on sort of the tripartite division. I think as we saw from the launch of GPT-5 last year, how underwhelmed most people were by GPT-5. And I think that that was a fascinating sort sociological episode in itself, particularly because if you look at it purely in terms of benchmarks, the line kept going up and continues to keep going up across most of the things we know how to measure.

There is no evidence of a slowdown in AI capabilities, at least in terms of evals and benchmarks. And yet people were a lot less impressed by GPT-5 than previous models. I think there are a few little just interesting reasons for that. A very basic one is just that cadence of release has massively increased. So it’s no longer the case that we’re waiting a year and a half between model releases with no releases at all in between. Now we get updates pushed every couple of months.

So there are going to be fewer wow moments. I think that sort of partly explains why maybe people were a little bit underwhelmed by GPT-5. Another fact is that I think, another sort of constraint on how impressed people are is that I think models are already good enough at most of the kind of tasks that most people use them for such that new releases don’t radically change people’s affordances with the system.

I mean, I think there are occasional specific domains in which they do. So just to give one example of, I think, a major transition, as it were, in capabilities. I think the release of NanoBanana, Gemini’s integrated image model, dramatically changed what you could do with images and data, specifically because NanoBanana is very good at threading text data and sort of semantic content with images. So for example, with NanoBanana, you can have a long conversation and say, now create an infographic or a mind map of the conversation we’ve just had. NanoBanana from day of release could do that in a way that every other previous image model would fail abysmally at. So there are these kind of like sudden, wow, here’s something new that I could do that I couldn’t do before. But in terms of just sort of general performance of language models, I think they are good enough for most purposes that there are fewer wow moments there.

I think more broadly regarding which of these three ways of measuring AI progress are most relevant and important. I think it depends on the domain. So I think there are isolated domains as measured by particular benchmarks, like some of the mathematics benchmarks, where those benchmarks do have immediate significance, right? If we are using AI models to solve near future, outstanding major problems in mathematics, right? Then I think benchmarks might be getting us pretty close to measuring the underlying criterion that we’re interested in.

Ultimately though, I think it’s the economic impacts that are most pressing and most exciting and most scary. But as you said, there’s so much more to those than just the raw capabilities of the system.

Dan Williams: Yeah. I do think though, I mean, now I’m just becoming a Dwarkesh fanboy, but just to observe another point that he made in this blog post, and we can link to this because I think it was a very interesting one. He says, people who make this point about the difference between, you know, the raw capabilities of the system and the rate of diffusion, or in other words, say that you can’t evaluate the capabilities of the system merely by looking at the degree to which it’s integrated into people’s workflow because that’s going to be slowed down by all sorts of factors. He says that’s basically a cope on the grounds that if we really had AGI, it would integrate incredibly quickly.

And I think an analogy he uses, if you think about, you know, immigrants integrate into the economy very, very quickly because they’ve got this wonderful, flexible, like general purpose intelligence that human beings have. And he says, well, if you really are imagining an AGI of the sort that people like Sam Altman and so on were forecasting, then you wouldn’t have all of this friction when it comes to integrating these systems into an organization’s workflow because they will be able to do everything that a human being can do just better. So it wouldn’t be any more difficult than integrating a human being into it.

It’s an interesting argument. I don’t know whether I’m sort of fully persuaded by it. Before we move on, did you want to respond to it?

Henry Shevlin: Yeah, I think it’s a fantastic argument. I think actually one useful, I think as we move towards greater and greater degrees of generality, the kind of existing structural constraints and the problems imposed by the jaggedness of models are going to become less pronounced. So I think that is a useful way to sort of measure progress towards AGI is thinking about the degree to which systems are capable of overcoming sort of external constraints, external limitations.

So, you know, for example, something as simple as a model being assigned a task, realizing it doesn’t have the internal resources to solve that task, identifying tools it could use so that it could solve that task, and then using those tools. I think that is the kind of behavior. I think we see some of that already with stuff like Claude Code and its current form. But I think that is a good way to think about what progress towards AGI looks like, overcoming the kind of structural constraints that may not be pure limitations of the model, but it has something important about the model if it can work in indirect ways to overcome them.

Dan Williams: Yeah, and I think it’s also interesting because once you start thinking in that way, it really does put pressure on this sort of AI as normal technology perspective that says, look, if you look at the history of technology, the process by which these technologies diffuse throughout the economy and throughout society more broadly, it takes a long time for all sorts of reasons. You might think, okay, but you can’t look at the history of previous technology because something like AGI would be a radically kind of sui generis technology precisely because it will be very easy to very quickly integrate into people’s workflow and into the sorts of things that companies and so on are doing.

Maybe just on this issue of these different ways of evaluating AI progress, before we move on, we could touch on two different things. The first is, so probably the most influential benchmark at the moment is this METR or MATR. I don’t know exactly how you pronounce it. Model Evaluation and Threat Research graph. I think to be honest, we’d have to do a whole separate episode on this where we really get into the weeds because I think the methodology and everything is very, very complicated. But basically, as I understand what METR are doing with this metric, which is basically a time horizon metric, is it saying, look, lots of other benchmarks, they’re evaluating AI’s ability to perform a set of say abstract cognitive intellectual tasks. But what we should really care about or at least one thing we should really care about if we’re interested in these things like agency and the ability to master context and this sort of constellation of abilities that seem to go along with agency is sort of how long it takes a human being who’s a professional to perform a task.

And to the extent that AI systems are getting better and better at performing tasks that would take human beings a very long time to do, that’s telling you something really kind of important about the capabilities of these systems and how fast they’re progressing. And as I understand their metric, basically what they’re saying is there’s been a kind of exponential growth such that the task length of tasks that AI systems can perform is doubling something like between every three or every seven months, where task length there is specified by how long it would take a person to perform that task.

So there’s a very nice quote from Roon, who’s a popular social media AI commentator. He says, the METR graph has become a load bearing institution on which our global stock markets depend. And the thought there is many people are looking at this graph and they’re seeing line go up. They’re seeing this rapid progress. They’re extrapolating that into the future. And that’s why there’s so much optimism about the capabilities of these systems and how they’re likely to develop into the future. That was my current understanding of this graph and the metric that it’s using. Do you have any thoughts about this evaluation?

Henry Shevlin: No, I think you did nothing, no major notes. I think you did a great job of describing it. So just a flag that sort of the METR time horizons task is specifically focused on software engineering tasks. So that is a slightly narrower set of tasks, but it’s one that obviously has massive economic value and is also potentially relevant if we’re thinking about any kind of recursive elements in AI development, you know, software engineering tasks are relevant to building AI systems. So to the extent that we’re finding massive time-saving improvements through the use of AI tools, that might be expected itself to accelerate AI development. So that’s another reason I think that this is so important, maybe less so for the stock markets than sort of the more kind of future-oriented predictions about where AI capabilities are going to go from here.

But of course, absolutely. This is probably the most interesting benchmark to watch when thinking about real world impacts of AI. Software engineers are a very, very expensive group of people to employ. And to the extent that AIs function as massive time savers in those tasks and can do more and more complex workflows within these tasks, that has massive real world impact and significance.

Dan Williams: Yeah, and I think this point about overwhelmingly the tasks are kind of software engineering tasks. So, you know, a software engineering task that might take a human being six hours to complete in and of itself, you know, that’s going to limit the generalizability of this metric because you might think lots of tasks within the world just don’t have the structure of software engineering tasks.

But I think there’s also a sort of, I mean, there are also all sorts of methodological questions about how they’re calculating this and so on. And like I say, I think we should do a separate episode where we sort of dig into this in detail. But I think there’s this other issue which is just to do with benchmarks as a whole. In order for us to be able to have graphs like this, we need tasks in which basically there’s a correct answer or a correct output.

I take it one worry here is just the kind of classic Goodhart’s. Is it Goodhart’s law? You know, when a measure becomes a target, it ceases to be a good measure. So a risk that with any given benchmark, we’re getting systems that are getting better and better at doing well on the test in ways that don’t necessarily correlate with the kinds of things that we really care about.

But I think there’s also another worry where even if you set that aside, the worry would be something like, okay, by the very nature of these benchmarks, where there’s a kind of clearly defined correct answer or output, you’re not tapping into the kinds of things that really matter to a lot of human intelligence, where it’s not a simple issue of here’s the finish line or here’s a clearly defined correct answer or correct output. And I mean, how far do you think that skepticism can go? Like if someone says, look, there’s a possibility here that even though we’re seeing rapid progress when it comes to these benchmarks, including this sort of time horizon benchmarks, which seems like it should be really informative. Nevertheless, it’s just not really telling us anything interesting about the sort of broader set of competencies that matter for real world deployment. Like how much skepticism do you think is tenable when it comes to the gap between benchmarks and sort of the capabilities that we really care about.

Henry Shevlin: Yeah, I think it’s a persistent worry all across not just AI research or even ML in the broader sense, but psychology. Criterion problems is sometimes called, shot all over the place. We have a dozen different ways of measuring creativity, which have minimal predictive validity for one another. As soon as you operationalize a really interesting target, you immediately lose many of the features that make it interesting in the first place. So I think it is an absolute legitimate worry.

That said, I think that I should be able to do better than this, just anecdotally, I think we are seeing models become just generally more useful. If they were improving in benchmarks, but that wasn’t translating into actual real world utility on different tasks, that would be a real red flag.

I can speak about your experience, but my experience is that basically every successive model release is at least somewhat better. I can do some new things with it. And that’s why I think the METR Time Horizons benchmark is a valuable one, but why I also think sort of more grounded economic benchmarks, for example, the degree of internal value created by different, by AI usage in different industries, the degree to which industries are successfully implementing AI automation projects and so forth, they’re an absolutely necessary complement because they’re measuring something that still has some criterion problems, like generating economic value, but is much more tangible and less likely to be a mere artifact of our sort of testing of our evaluation framework.

Dan Williams: Yeah, okay, great. Let’s, I think there are two things to kind of finish on. One of them, I think we can be brief because we’ve really kind of already touched on this, but it’s what we should expect the transition to a post-AGI world to be like, however you understand AGI. And the other is predictions for 2026 in terms of how we see these capabilities.

But first, just want to give you an opportunity to have a take on Claude Code. So, I’m sure you’ve also seen a lot of commentary, a lot of buzz, a lot of discourse to the effect that Claude Code, and just for those who aren’t really in the weeds in AI, Anthropic is a frontier cutting edge AI company. They’ve got a model called Claude. And as part of that, they’ve got Claude Code, which is primarily used for sort of software engineers and coders, but apparently it has much broader application.

I should say I haven’t used Claude Code. I do use Claude all of the time, which I think is incredibly impressive. I haven’t used Claude Code. I’m very, very skeptical that it’s AGI or if it is AGI, I think that probably tells us that the concept of AGI can’t do the work that many people have assumed that it can do. Have you got a take on Claude Code before we move on?

Henry Shevlin: So I haven’t played around with it as much as I would have liked. And it is, I think, one of the more daunting models for non-technical people to use. Even installing it, for many people, will be a little bit of an adventure. But particularly speaking to friends in technical whose jobs are primarily technical, the wow factor seems to be huge on the current iteration of Claude Code.

People are talking about how it’s transforming their workflows, enabling them to do a whole suite of tasks they couldn’t have dreamt of doing before. And I do think it is a significant landmark. Yeah, I think it probably is a taste of the kind of capabilities that we’re gonna see over the course of the rest of this decade, where it’s not just people slotting AI to do specific tasks or sub tasks within their own workflows, but being able to delegate whole workflows to Agile Systems.

Dan Williams: Yeah, okay. And that in a way that leads us onto the first of those two points that I wanted to end on, which is how we should think of the transition to a sort of post-AGI world. I mean, I take it there’s a model you sometimes come across where it’s almost like it’s the atom bomb going off in the Manhattan Project. You reach something called AGI and it’s just radically transformative immediately, for various reasons, maybe because of the capacity to take AGI and use it for large scale automation, but also potentially because of the ability of AGI to get involved in the AI R&D process, triggering this kind of intelligence explosion.

I’m really skeptical that that’s the right way to think about it. I think what we’re seeing basically is kind of incremental improvements in the capabilities of these systems when it comes to things like agency, multi-step sort of long time horizon planning, continual learning and so on. I don’t think there’s gonna be like a big bang. I think we’re gonna see this sort of incremental progress where if you compare, you know, one year to three years down the line, it will seem like this huge disparity, but living through it, I think it’s gonna seem very continuous.

And also when it comes to the impact of this kind of technology on the economy for the reasons that we’ve got into. I think there are going to be all sorts of bottlenecks. There’s going to be so much opposition, even when you’ve got capabilities that are very powerful, integrating it into people’s workflow and so on and so forth. So I’m definitely not really expecting a kind of big bang here. And I think people saying that with highly agentic, at least relative to what’s come before AI like Claude Code, you’re seeing kind of baby AGI, I think that might be true to an extent relative to certain understandings of what AGI is, but that just tells us, I think, that AGI isn’t going to be this landmark event. It’s going to be a sort of continuous incremental improvement across lots of different capabilities. So that’s my high level take. Do you have a different take? Do you have to build on that in any way?

Henry Shevlin: Yeah, I think I largely agree with your take that we’re not going to have a sort Trinity test equivalent moment if we’re going to use the analogy of the Manhattan Project, right? There’s not going to be a sudden moment where a lab says we’ve built AGI. Instead, it’ll seem very incremental and continuous to most people, even those who are following what’s happening. And then by the end of this decade, we’ll look back and say, holy shit, how far we’ve come.

And I think I don’t think there’s any reason to think that progress towards the kind of highly general autonomous systems that, or highly general autonomous capabilities that people associate with AGI. I don’t think there’s any reason to think that that progress isn’t continuing the pace. And I do think, to go back to Claude Code, that it is an example of the kinds of really consequential leaps that we’ll see.

So Ethan Mollick today has a, Ethan Mollick has a piece, brand new piece called Claude Code and What Comes Next, where he talks about using Claude Code to generate a passive income and how it creates hundreds of files for him. Having worked autonomously for 74 minutes, it deploys a functional website that could actually take payments. It got various things wrong along the way, but it was a far cry from sort of the Claudius vending machine experiments from earlier this year, earlier last year.

So yeah, I think we’re going to look back at the end of this decade and realize how far we’ve come, but there’s not going to be a single Trinity test style moment. And I think an interesting parallel here is actually with the Turing test. I think a lot of people were expecting the Turing test to be, there would be like a decisive moment where it’s like, wow, computers can now pass the Turing test. I don’t think that would have been a smart thing to think because Turing’s original test is woefully under-specified. He doesn’t sort of give exact time windows and so forth, and there are various constraints you can build in.

But I think at this point now, the Turing test is no longer especially relevant as a measure of AI capabilities. It’s still of interest, but it’s no longer the case that it’s sort of a clear benchmark we’re working towards. We have had multiple instantiations of the Turing test now that show frontier AI systems can fool humans over two or five minute time horizons with basically at 100% success rate, like where humans are chance at guessing whether they’re talking to an AI system or a fellow human.

But it’s like that benchmark slowly faded into the background rather than being a decisive moment. And I think AGI is going to be very similar. By the end of this decade, I do expect that we will have autonomous, agentic AI systems deployed in pretty much every industry. The vast majority of people’s workflows and daily jobs are going to be very, very different. I don’t think by the end of this decade, for what it’s worth, that we’re going to be looking at mass unemployment.

I actually quite like Noah Smith’s, I don’t fully agree with it, but Noah Smith has this sort of piece on how even in an AGI world, we might still have full employment, leveraging this sort of concept of comparative advantage, the idea that there are always going to be things where it’s cheaper to employ or easier to employ a human to do a given task. And I think that’s going to be one of the things that prevents sort of mass technological unemployment. Also things just like compliance and the fact that you’re going to need to have humans on the loop in many tasks, monitoring AI systems to ensure that you’re abiding by regulations. But I do fully expect the increasingly general AI systems about which will be debates around AGI will seem increasingly irrelevant to be ubiquitous by the end of this decade.

Dan Williams: That’s a nice prediction. Yeah. The prediction concerning AGI is that debates around AGI will go the way of debates about the Turing test. Also, just to add to that point you made about the economics of this, I think the comparative advantage point is very interesting and I think it’s very important.

There’s also a kind of obvious thing which sometimes gets missed in questions about automation, which is when we say, you know, AI systems that can, let’s say, outcompete human beings doing what human beings do. Really, the contrast there is outcompete human beings using AI systems. So it’s not like human intelligence is this fixed target, such that we need to build AI systems that can outcompete human beings as they are in 2026. Human intelligence in general depends on all sorts of technological scaffolding and so on. And that makes it a moving target.

I certainly find in my own work, me with AI, so much more productive and effective than me without it. So if you ask, you know, could AI systems beat Dan without using AI? That’s a very different question, I think, than could you design autonomous, you know, flexible, continual learning based AI systems that could outcompete me with access to those systems? And I think that’s also got sort of implications for how we think about the real world impact of all of this.

Henry Shevlin: Yeah, I guess I do want to, before we go into predictions, I just want to add one final sort of coda here, which is I’ve, in that sort of foregoing prediction of what we’ll see this decade is not to be extrapolated outwards. I think there may well be a point probably beyond the point of this decade where things start to get really, really weird, where sort of the degree of the absolute advantage systems have really just fundamentally starts to reshape workflows and value chains in a way where human labor may eventually, and maybe in some point in 2030s, start to struggle to fit in.

So I’m thinking here of this wonderful piece, very far-sighted flash fiction piece by Ted Chiang called Catching Crumbs from the Table, published in Nature Futures a long time ago, about 20 years ago, where he talks about post-human science and the idea that eventually science reaches the point where it can only be done by AI systems. And just because the kinds of theorems, kind of tools being used are just incomprehensible to humans. And he imagines this sort of cottage industry of sort of explainers where humans try and understand, you know, we’ve built this, AI has developed this new alloy that we are completely incapable of understanding in terms of our existing material science, but let’s do our best, right?

So I think it is possible, I think broadly plausible that if we extrapolate far enough outwards that we might start to hit that point. And then I really do think all bets are off. What does human employment in finance look like when you have super intelligent financial managers supervising super intelligent analysts? What is the role for the human there? Is it just going to be that you sit next to your mainframe running 10,000 super intelligent AI finance agents, and if they ever do anything illegal, you get fired. That might be what sort of people’s jobs start to look like at that point. But I think that’s slightly longer time horizons instead. What I see over the course of this decade is not mass unemployment, but definitely radical changes in human workflows.

Dan Williams: I also think like one of the things that at least in my own case, the reason why I find this so difficult to think about is I just don’t know what to make of the intelligence explosion argument. And I feel like the people that are expecting a real sort of discontinuous leap here, at least relative to human timeframes, they’re imagining a process which will be incredibly rapid precisely because of this model about recursive self-improvement.

So Will MacAskill and Fin Moorhouse have a really nice article on the intelligence explosion and how you could basically compress sort of a hundred years of technological progress into, you know, much, much shorter timeframe. And if that kind of analysis of what you might see with this intelligence explosion, as they understand it, is correct, then, you know, my current view that a lot of this is going to be sort of relatively incremental and continuous and there aren’t going to be any sharp breaks might just break down entirely. I feel like I need to get a good grip on what to think about that whole argument. But we’re going to be devoting episodes this year to people that—

Henry Shevlin: Just to tee up one thought here. So on this point, a sort of prologue or a preview of the future episode, I do think for all of the worries, I think in many cases legitimate worries about hype in Silicon Valley, I think that is absolutely a legitimate cause of concern. And the quasi-religious nature, I think Karen Hao was talking about, you know, how there often is a quasi-religious element to some of these predictions. I think that’s absolutely right. I don’t think that’s disqualifying, right? But I think absolutely, if you don’t think there’s something, there’s a religious element in a lot of talk about AI and AGI in particular, then you’re not paying attention. There absolutely is.

So I think that’s true, but there’s also a bias in the opposite direction, normalcy bias, and the meme version of this is nothing ever happens. But I think if you just look at the recent history of our species, we have many discontinuities, whether that’s the Industrial Revolution, the Agricultural Revolution, even biologically, the emergence of multicellular life in the Ediacaran and the Cambrian explosion, right? The history of life on earth and the history of human civilization is full of these sort of major transitions, these relatively rapid discontinuities. So I think the assumption that sort of nothing ever happens or that, you know, things are basically going to tick on as normal is another bias that we need to be wary of.

Dan Williams: Completely agree. Okay, you said predictions end of decade. How about this year? So when we do this conversation at the beginning of 2027, here’s maybe one way of thinking about it, right? Capability predictions is what I think you expect these systems to be able to do by the end of this year that they can’t do now. Economic predictions. And here I think the central question is, is there a financial bubble here which is going to burst and, you know, potentially even initiate another AI winter, you know, a period in which lots of the enthusiasm dissipates and then maybe, you know, political predictions, right? At the moment, I think people are not aware of what’s coming. People generally are pretty hostile towards AI and pretty, pretty fearful of it, but we haven’t really seen kind of coordinated political movements against AI, where that’s a defining issue. Should we expect to have seen that by the end of the year?

Henry Shevlin: Oh, so many tricky questions. You know the famous quote, it’s hard to make predictions, especially about the future. I want to say in some ways it’s almost harder to make predictions about the near term than the long term, right? Insofar as those predictions have to be more fine grained, more falsifiable. You know, we can say, oh yeah, like by 2030 things will be really different. That’s easy, right? Saying like what’s going to be different by end of 2026 in some ways is harder.

So I don’t expect any massive AI bubbles. I think as the industry matures, like we will hear about various midsize AI companies who’ve been selling vaporware, going bankrupt. And I think the usual suspects will sort of call this out and say, aha, you see there’s an AI bubble all along, but I don’t expect it to be an industry wide trend. I don’t even expect it to be a major bubble in sort of frontier LLMs or frontier model developments.

But the other thing I just emphasise on bubbles when people talk about the AI bubble is that AI is rapidly proliferating into a whole bunch of different things. So like driverless cars, for example, which were, you know, often decried as vaporware in the 2010s, they’re absolutely here now. You can take a Waymo in San Francisco and several other cities today. And 2026 is one of the big years for rollouts of driverless cars. You now have Waymos in London, right? And I think there’s some like 30 cities globally introducing driverless car pilots over the course of this year.

So even if it turns out that OpenAI hit a wall or there’s some major scandal or they’re over leveraged, none of which I think is true. But if that did turn out to be the case, it wouldn’t kill AI in the same way that previous sort of AI winters have sort of killed research in AI, not quite across the board, but more broadly. At this point, everything from driverless cars to autonomous weapons systems to AI and medical research, AI and material sciences to AI for a wide range of tasks. I think it’s too diffuse and too plural for any kind of single bubble events to kill the industry as a whole.

But yeah, I also don’t expect any kind of, even a bubble in the domain of language models. So that’s one point.

Another area where I do think in terms of economic impacts, I think those will grow. I think more and more people are gonna be start, gonna be seeing impacts of AI in their workflows. It wouldn’t surprise me if we start to see some big legacy companies really, really struggling because they’re being outcompeted by startups or scale-ups that make better use of AI than them.

I think more and more companies are going to have to face this difficult challenge of, do we go all in on AI at this point, or do we still try and manage a slow transition? So I think it’s going to be a very economically disruptive year ahead. I think part of the reason for that is I think 2026 really will be the year of agents. So a lot of people, I think Sam Altman said 2025 was going to be the year of agents, but I think that was premature. AI, agentic capabilities of AI understood here as sort of their ability to do long-term complex multi-step tasks is only really getting going. But I think particularly as we start to see more deployment of these AI agents that are in turn generating useful training data about what works and what doesn’t, I think we’ll start seeing more and more really valuable AI agentic products over the course of 2026. I think Claude Code is very much a taste of what’s to come. So big economic disruptions by the end of 2026.

And I think to touch on your political point, I think this is going to lead to increasing backlash. A really interesting phenomenon at the moment is that on the right in America, there is a relatively unified or at least superficially unified pro-AI mood. I think a lot of this has to do with the influence of, you know, a lot of big or the alignment of a lot of big tech with the Trump administration, which has its own reasons for being very pro-AI, geopolitical considerations and so forth. But I think one interesting prediction would be that the right wing on the American right in particular, maybe the global right, pro-AI attitude may start to break down.

I think we’re seeing some trends, some signs of this in domains like AI in young people. There’s an increasing number of sort of Republican politicians who are very, very concerned about things like LLM psychosis, about appropriateness of content that minors are accessing, about impact on youth mental health.

And I think we might actually start to see, that’s one of the areas where we might start to see in the US context, some bipartisan consensus emerging on the need for AI regulation. I’d say partly that’s maybe due to things like family values, protecting young people being values that are as central, if not more central to the right than on the left. So it’s inherently bipartisan, but also because the idea of better regulations around protecting young people don’t necessarily interfere with kind of geopolitical applications of AI. You know, strict rules on under 18s using ChatGPT is not going to prevent the US from using AI tools effectively in future military conflicts and so forth. So that’s one political development.

I think at the cultural level, things are just going to get weirder and weirder. So, you know, we’ve done two episodes on social AI. I think 2026 social AI is going to continue to become more and more ubiquitous.

Sadly, I think we will see many more New York Times and legacy media stories about LLM psychosis, LLM exacerbated or triggered or implicated suicides. I think we’re going to continue to see deep entanglements, deep relationships between humans and AI systems become more and more common. And maybe an outside prediction, I do think the kind of AI welfare, robot rights movement is going to continue to gather steam. Probably not a major culture wars issue, even by sort of this time next year. But I think it’ll, you know, it will go from being this, I mean, it’s no longer even that niche, but still relatively niche thing worked on by a few think tanks to something that is increasingly something the general public are thinking about.

Dan Williams: Great stuff, great stuff. I think many of my predictions, to be honest, overlap with yours and probably a unifying theme is I expect all of these things to happen kind of gradually. So I think the systems will get better and better, not just in terms of how they perform on benchmarks, but in terms of their capabilities. But I don’t think we’re going to be seeing like a big bang upgrade this year.

On the stock market, I mean, there I think there might be a financial bubble that bursts, even though I take your point that AI itself as a technology is not going away. And I think people often conflate those two things in an important sense that they’re kind of orthogonal in the sense that it could be the case, and I think it definitely will be the case, that AI becomes increasingly impressive, capable, and integrated into the economy and into society more broadly. It could also be the case that given the financials of many of these companies and investment decisions, et cetera, et cetera, that you see some quite significant bursting of the bubble that has short-term significant economic impact. I’m probably kind of 50-50 on that, and I just don’t feel like I’ve got the expertise to really evaluate it.

All of the other things, I think you’re sort of directionally correct as they say. I do think one thing where again, I’m probably 50-50 is my understanding of all of the big AI sort of frontier labs at the moment is a major focus is on this kind of continual learning, building advanced AI systems of the sort that we’ve got today that can engage in continual kind of experience-based learning. So you don’t have to construct kind of bespoke reinforcement learning environments for specific tasks, but you can drop a system into an environment and it will be able to update its weights sort of continuously as it engages with that environment in much the same way that human beings and other animals can do.

Given that, at least to me as an observer, it seems like there’s so much focus on that and a recognition that that will be a really big change. I suspect that this time next year, maybe I’m sort of 50-50 here, that we will have seen at least one AI lab that’s made some significant progress on that. I don’t think it will be kind of immediate now they can do it, but I think maybe there’ll be a paper that’s released. Maybe there’ll be a kind of updated model that can do some version of this. And that would be a really huge, I think, story in terms of the historical development of these technologies. Other than that, I think I basically just agree with you. Directionally—

Henry Shevlin: Yeah, directionally correct is the best kind of correct. I know, yeah, I think there’s only not too much disagreement there between us. I mean, just to throw in one final thought, I wouldn’t be surprised, despite everything we’ve said, if AI is not the biggest story of this year. I think we live in an exceptionally unstable time, probably the most unstable time of my entire lifetime. And I wouldn’t surprise me at all if geopolitics or, I mean, particularly geopolitics, but potentially other domains create bigger surprises that swamp the relevance of AI, whether that’s war in the South China Sea, a major break between Europe and the US.

And that, you know, we’re focused here on AI, but I think that will have very big implications potentially for AI just because AI supply chains are so delicate. You know, a war in the South China Sea, for example, I think could be one of the biggest spoilers for most people’s AI timelines. So despite all my excitement around AI, I think given the sheer instability in the world right now, it may not end up being the biggest story of 2026.

Dan Williams: We are cursed to live in interesting times. Okay, that was such a fun conversation. We’ll see everyone in a couple of weeks.

I started this blog on January 1^st, 2024, so I’ve now been publishing weekly essays here for over two years. It was one of the best decisions I’ve ever made. I’m grateful to everyone who reads and engages. Even the haters and losers (of which, happily, there aren’t many) often provide interesting and informative critiques.

I’m especially thankful to those who have paid subscriptions. I’m aware that many of you subscribe not simply to access paywalled articles but to support my writing. I’m truly moved by this. It’s also a helpful corrective to my broadly cynical views about human nature.

As of 5th January 2026, the blog has roughly 19,800 subscribers. It averages approximately 120,000 views per month, though with substantial variance.

In this post, I will review the blog’s output from 2025, recommend the best things I read last year (as well as other favourites), and then briefly outline how I will approach this newsletter in 2026.

Year in Review

Based on the number of readers, here were my top ten essays in 2025:

1. Status, Class, and The Crisis of Expertise — This argues that one underappreciated factor driving the “crisis of expertise”, and hostility towards knowledge-producing institutions more broadly, is feelings of humiliation and resentment among conservative voters with low levels of education, who view experts as a condescending and hostile social class. Among many others, it draws on the work of Thorstein Veblen (whose concept of conspicuous consumption inspires this blog’s title), Marcel Mauss, Will Storr, Musa al-Gharbi, David Hopkins, and Matt Grossman.

2. Let’s Not Bring Back The Gatekeepers — This argues that the media transformations of the digital age have created new pressures and responsibilities for small “l” liberals like me. Put simply, if you can no longer control the public conversation, you must participate in it, which, especially in recent years, too many liberals have been unwilling to do.

3. Is Social Media Destroying Democracy—Or Giving It To Us Good And Hard? — Much of the discourse about how social media is terrible blames engagement-maximising algorithms. Because companies profit by keeping people engaged and glued to their screens, algorithms feed people the epistemic equivalent of junk food: content that generates outrage and resentment, inflames our tribal instincts, and taps into negativity bias. Although important, I argue that a bigger factor is simply that social media has radically democratised media. Many people have ugly, illiberal, misinformed, and generally bad views and values, and social media gives them a platform and much greater consumer power. Admittedly, this view is not very politically correct to acknowledge, but it’s accurate.

4. On Highbrow Misinformation — There’s a tendency to think that “misinformation” is entirely something that right-wing elites, sinister corporations, and uneducated hoi polloi engage in. But in fact, there is a considerable amount of left-coded “highbrow misinformation” that circulates within the prestigious knowledge-producing institutions that bang on about the dangers of misinformation. I give many examples in this essay and also explain why and how such misleading content emerges and proliferates, often as a consequence of the politicisation and progressive groupthink that has captured many institutions.

5. The Case Against Social Media is Weaker Than You Think — This essay summarises and develops ideas from an article I wrote for Asterisk magazine (“Scapegoating the Algorithm”). The main point I make is that although social media platforms obviously aren’t harmless (see Essays 3 and 4), most of the discourse surrounding their dangers is driven more by vibes, anecdotes, and moral panic than rigorous argument or social science.

6. The “Everyone is Biased” Bias — This essay makes the simple point that although everyone is biased in ways that are important and under-appreciated, it’s not the case that everyone is equally biased. There are significant differences between individuals, norm-governed communities, and institutions in how they handle and process information. So, a recognition of the universality of bias must co-exist with avoidance of the “everyone is biased” bias, which flattens such important differences.

7. The World Outside and The Pictures in Our Heads — This provides an opinionated summary of the Lippmann–Dewey debate over democracy, public opinion, and the role of experts in complex, modern societies. I am a huge Walter Lippmann fan. I think he’s the most insightful political epistemologist of all time. This essay sets out his views on the essentially insurmountable challenges of acquiring adequate political knowledge and understanding in the modern world.

8. On Conspiracy Theories of Ignorance — This essay explores Karl Popper’s critique of the “conspiracy theory of ignorance,” which assumes that the truth is so self-evident that popular false beliefs must result from some deliberate conspiracy. Although Popper was mostly concerned with how Marxists and other leftist intellectuals think about “ideology”, the critique is equally pressing for much establishment hysteria about “disinformation” and “merchants of doubt” as the source of all popular misperceptions. I try to explain why Popper’s critique is valuable even though the world does in fact contain highly consequential conspiracy theories of ignorance.

9. On Becoming Less Left-Wing (Part 2) — This is the second in my series of essays detailing how I have become less left-wing in recent years. I explain in greater depth than I have elsewhere why political knowledge is, in general, extremely hard to attain, how tribal allegiances and other interests inevitably distort our beliefs, and why political ideologies are both inevitable and inevitably simplistic, selective, and vulnerable to distinctive failure modes. Think of it as “postmodernism but good”.

10. Domination and Reputation Management — A popular theory of “system-justifying ideologies”—for example, the belief in the divine right of kings, or that group-based domination is legitimate because subordinate groups are intellectually and morally deficient—is that they function to persuade the oppressed to acquiesce in their oppression. I argue that the real function of such ideologies lies in reputation management among oppressors. This leads me to a broader account of how reputation management doesn’t just produce apologetics for power; it also distorts the belief systems of those who think they’re “unmasking” power, including many “radical” left-wing intellectuals whose critiques of “ideology” were easily co-opted by history’s most despotic, exploitative regimes.

There are several unifying ideas across these essays:

• The truth is not self-evident, even though we are often disposed to think that it is. Reality is vast and complex, much more complex than we can even imagine, and we access it not directly but through messy, often-opaque chains of testimony, trust, categorisation, and interpretation. Even the part of reality that we are in “direct” contact with—the bits we can actually perceive—are typically understood through socially-learned conceptual schemes and belief systems. As Walter Lippmann put it, modern politics deals with “indirect, unseen, and puzzling facts, and there is nothing obvious about them.”

• Experts are necessary but human. Although journalists, pundits, intellectuals, scientists, and other “epistemic elites” have critical advantages in confronting and uncovering such facts, they are also vulnerable to the same biases as everyone else. Moreover, their advantages are often used to indulge such biases rather than correct them. The critical theorist who “unmasks” ideology doesn’t escape ideology. “Misinformation experts” aren’t strangers to misinformation. And so on.

• The epistemic is not merely epistemic. The beliefs, narratives, ideologies, and social norms that regulate our minds and behaviour are distorted by propaganda, grubby motives (e.g., self-interest, reputation management, and status competition), and tribal allegiances. Such distortions are obvious in our rivals and enemies but not in our friends or ourselves. The failure to correct for this bias produces lots of bad social theory and politics.

• Humans are kinda sorta rational. The popular image of human beings as credulous fools riddled with cognitive biases is mistaken. We are far from perfectly rational, of course, but people—yes, even the people you dislike—are typically far more sophisticated, critical, and intelligent than they seem. The contrary impression arises from a combination of the “third-person effect”, misunderstanding people’s real goals (e.g., assuming their primary motivation is always to figure out the truth), and underestimating the challenges of acquiring knowledge in complex, modern societies (see above).

• Avoid technopanics. Technology is highly consequential, but most popular (and much scholarly) discourse about technology involves simplistic moral panics that obscure the complex, sophisticated ways people use such technologies, and their interaction with pre-existing features of societies. People aren’t passive, credulous victims of algorithms. And the effects of social media platforms are often mediated by long-standing pathologies of democracy, public opinion, polarisation, the growing diploma divide, and the politicisation of institutions, many of which are far more complex and uncomfortable to discuss than algorithms and Russian bots.

Podcasting

I appeared on several podcasts this year, including Evolutionary Psychology and The Good Fight with Yascha Mounk. Both provided really valuable outlets for discussing my views about human nature, belief, and self-deception (in the former) and misinformation, institutions, social epistemology, and politics (in the latter).

In the last few months of the year, I also started an AI podcast with my friend, Henry Shevlin, where we discuss the big-picture philosophical, scientific, and political questions thrown up by rapid developments in artificial intelligence.

I am convinced that AI will be utterly transformative in the coming years and decades. Although I did my PhD (between 2015 and 2018) on various philosophical questions surrounding generative AI, I immediately pivoted to the area of “political epistemology” in the years that followed, albeit still with a strong focus on psychology and cognitive science in a way that distinguishes me from most scholars in this area.

Now, I am back to thinking about AI a lot, focusing less on the technology itself than on its social and political significance (including its interaction with questions concerning misinformation, institutional trust, expertise, and public opinion).

My podcast with Henry is a way to keep up to date with this area in ways that other people will hopefully find beneficial. In the first six episodes, we covered big-picture debates about AI and existential risk, consciousness, education, LLMs’ environmental impact, and relationships:

1. AI Sessions #1: AI – A Normal Technology or a Superintelligent Alien Species?

2. AI Sessions #2: Artificial Intelligence and Consciousness – A Deep Dive

3. AI Sessions #3: The Truth About AI and the Environment

4. AI Sessions #4: The Social AI Revolution – Friendship, Romance, and the Future of Human Connection

5. AI Sessions #5: How AI Broke Education

6. AI Sessions #6: AI Companions and Consciousness

I will always be a writer first and foremost—that’s where my strengths lie—but I’ve found these conversations to be really enjoyable and stimulating. This year, we will be speaking to many interesting guests.

Recommendations

2025 was an excellent year for Substack. I spend more time reading articles on this platform than on any other. For those (like me) interested in science, philosophy, and intellectually serious, evidence-based contributions to politics and current affairs, there is nowhere better.

I am reluctant to name specific Substackers I enjoy because I know I’ll accidentally leave out many brilliant ones. But if you want suggestions on whom to read, you can check my Recommendations, and also follow me on Notes in the Substack app, where I do my best every day to share the excellent articles I come across.

I read fewer new books than usual this year. The main reason is that I’ve been re-reading extensively with LLMs such as ChatGPT, Claude, and Gemini.

Book quality is extremely heavy-tailed. Most books are bad. A tiny number are exceptional. So, the overall value you get from reading is heavily influenced by decisions about what to read, and you are often much better off trying to master and internalise the ideas of exceptional books than reading new ones.

LLMs make this a lot easier. You can upload a PDF of the book and have a quasi-conversation with it, testing your understanding, receiving tailored explanations and tutoring, creating flashcards to import into programs like Anki (for spaced-repetition-based learning), and more. If you haven’t played around with NotebookLM yet, you’re making a huge mistake. So, this year, I spent much of the time I would have ordinarily spent reading new books on implementing this process for the best books I’ve already read.

Nevertheless, I did read some new books. More precisely, I read several new and several old books for the first time. In no particular order, here were the best ones:

Read more

The philosopher Jason Stanley has recently “fled” the USA, which he views as an authoritarian state undergoing a coup by a fascist party using Nazi tactics. In an interview outlining this perspective, he has harsh words for those who draw on the concept of “polarisation” to understand these developments:

“All the people talking about polarization are just fascism enablers. They’re almost worse than the fascists because they’re just like, “Hey, how do I keep getting money in power?” I’ll say the fascists are normal.”

Talk of polarisation is so objectionable, he says,

“because one side is led by fascists. I mean, it’s like saying the… problem with the Civil War was polarization. It’s literally like that… One group thinks that slavery is good, and the other group thinks it’s bad, terribly polarized. Or Nazi Germany. One group thinks Jews should be killed, the other one thinks they’re okay, it’s polarized. It’s nonsensical. It’s just fascism enabling.”

These sentiments are highly influential on the left, where articles have proliferated with titles like “The Problem Isn’t ‘Polarization’ – It’s Right-Wing Radicalization”, and “Our Problem isn’t Polarization. It’s Fascism.”

The Polarisation Industry

Such critics are responding to a large body of recent scholarship and commentary that links many of the world’s political problems to the extent of division, or “polarisation,” within and between societies.

Much of this discourse focuses on the USA, where Republicans and Democrats famously dislike each other much more than they did a few decades ago. For instance, between 2014 and 2024, the share of Democrats who reported that they would be unhappy if a family member married a Republican rose from 19% to 39%. For Republicans asked about Democrats, it increased from 22% to 33%. Of course, one also finds intense polarisation in many other contexts, ranging from Northern Ireland to Lebanon, the Israel-Palestine conflict to the left/right divide that structures democratic politics in many countries.

Many people view polarisation—especially “affective” polarisation, the fear and dislike of opposing groups—as a powerful force that threatens democracy, social trust, cooperation, and fact-based political debate and public opinion. In highly polarised societies, groups become less willing to compromise and transfer power, more willing to endorse political violence, and more likely to succumb to “tribal” or “sectarian” biases that distort perceptions of reality.

Terms like “tribalism” and “sectarianism” here underscore something important: the attitudes and emotions in highly polarised contexts like the US are not specific to those contexts. They emerge whenever intense intergroup conflict maps onto social identities such as partisanship, ideology, religion, sect, ethnicity, region, or tribe. That is, while every group rationalises their fear and hatred of the outgroup by pointing to its specific crimes, the emotions are strikingly similar and symmetrical across radically different conflicts, whether between Protestants and Catholics, Hutus and Tutsis, or Sunnis and Shias.

This pattern is typically explained in terms of our evolved “tribal” or “coalitional” nature. Our species was forged under selection pressures that favoured powerful motives and abilities for forming alliances designed to outcompete other alliances for prestige, dominance, and resources. So, when we support and identify with a group, automatic “coalitional instincts” are activated. We divide the world into ingroup and outgroup, us and them. We frame group relations as zero-sum conflict for power and esteem. We become obsessed with sending and monitoring signals of group loyalty. And we become instinctive apparatchiks and propagandists, embracing narratives crafted to make our side and its defining narratives look good, and the other side look bad, if not outright demonic.

Polarisation exacerbates those instincts, which in turn exacerbate polarisation, fuelling a runaway process in which competing tribes lose access to a shared reality and a willingness to empathise and compromise with each other.

The Critique

For Stanley and many others on the left, this diagnosis of modern politics is preposterous. Their central objection is that ‘polarisation’ implies a false symmetry, depicting two poles drifting away from a virtuous centre. This means that it can’t capture what the critics take to be a self-evident fact: that the real threat to liberal democracy and social justice comes from the right, the political home of extremism, racism, sexism, transphobia, lies, conspiracy theorising, and—at least in the views of figures like Stanley—fascism.

The problem isn’t that people are divided and tribal. The problem is that one tribe is a sinister menace to society. Treating that menace as an existential, fascistic threat doesn’t involve an irrational “tribal” psychology, an unfortunate hangover from our primitive, evolutionary past. It means seeing the far right for what it is. When confronted with this reality, the appropriate response is not depolarisation (i.e., political moderation); it is to be even more opposed—and hence more polarised—against it.

By misrepresenting this state of affairs and misdirecting our political energy, those talking about the dangers of polarisation and tribalism are complicit in normalising and enabling the right’s attacks on democracy and vulnerable minorities. As Noah Berlatsky puts it,

“A social science that sees polarization and partisanship as the main threats to democracy is a social science that implicitly—and often more than implicitly—is calling for white, Christofascist solidarity against Black (and feminist, and queer, and disabled) demands for justice.”

Am I A Fascist Enabler?

Read more

Rose is a PhD candidate in Psychology and Social Policy at Princeton University and a National Science Foundation Graduate Research Fellow. She conducts research on the social impacts of conversational AI agents like chatbots, digital voice assistants, and social robots. As founder of Ethicom, Rose consults on prosocial AI design and provides public resources to enable people to be more informed, responsible, and ethical users and developers of AI technologies. She is also co-host of the podcast, Our Lives With Bots, which covers the psychology and ethics of human-AI interaction now and in the future. Find out about her really interesting research here.

You can find the first conversation that Henry and I had about Social AI here.

Transcript

Note: this transcript is AI-generated and may feature mistakes.

Henry Shevlin (00:01)

Hi everyone and welcome to the festive edition of Conspicuous Cognitions AI Sessions. We’re here with myself, Henry Shevlin, my colleague Dan Williams and our guest today, Rose Guingrich, who we’re very lucky to have on the show to be talking about social AI and AI companions with us. We did do an episode on this two episodes ago, which featured me and Dan chatting about the rising phenomenon of social AI. And so if anyone wants a basic sort of primer on the topic, go back and listen to that as well. But today we’re going to be diving into some of the more empirical issues and looking at Rose’s work on this topic.

So try to imagine a house that’s not a home. Try to imagine a Christmas all alone and then be reassured that you don’t have to spend Christmas all alone. In fact, nobody ever needs to spend Christmas alone ever again because their AI girlfriend, boyfriend and B friend, husband or wife will be there to warm the cockles of their heart throughout the festive season with AI generated banter and therapy. Or at least this is what the promise of social AI might seem to hold. And in fact, just in today’s Guardian here in the UK, we saw an announcement that a third of UK citizens have used AI for emotional support. Really striking findings.

So cheesy intro out of the way. Rose, it’s great to have you on the show. Tell us a little bit about where you think the current sort of social AI companion landscape is at right now and what the major sort of trends and use patterns you’re seeing are.

Rose E. Guingrich (01:36)

So right now it appears as though we are moving toward an AI companion world where people are less judgmental about people using AI companions. It’s much less stigmatized than it was a couple of years ago. And now, of course, we’re seeing reports where, for example, three quarters of U.S. teens have used AI companions and about half are regular users and 13% are daily users. And so we’re seeing this influx of AI companion use from young people and also children as well, of course, from the reports that we’ve seen about teens using AI as a companion.

And I think looking forward, we’re only going to see more and more use of AI companions as companies recognize that the market is ready for these sorts of machines to come into their lives as these social interaction partners. And then if you look even further forward, these chatbot companions are going to soon transition into robot companions. And so there we’re going to see even more social impacts, I think, based on embodied conversational agents.

Dan Williams (02:46)

Can I just ask a quick follow up about that, Rose? So you said that this is becoming kind of more prevalent, the use of these AI companions. You also said it’s becoming less stigmatized. Do we have good data on that? Do we have data in terms of which populations are stigmatizing this kind of activity more or less?

Rose E. Guingrich (03:06)

So in terms of the stigma, we don’t have a lot of information about that. But we can look at, for example, a study that I ran in 2023 where I looked at people’s perceptions of AI companions, both from those who were users of the companion chatbot Replica and those who were non-users from the US and the UK. And the non-users perceptions of AI companions and people who use AI companions at that time was fairly negative. So for example, non-users indicated that it’s a sad world we live in if these things are for real. These AI companions are for people who are social outcasts or lonely or can’t have real friends.

And now in the media at least, we see a lot more discourse on AI companions and sharing about having AI companions. And one thing I can point to are subreddits. For example, My Boyfriend Is AI that has 70,000 companions. It is explicitly labeled as companions, whereas other subreddits are weekly visitors, visitors, users. This is companions and people on the subreddit are talking about their AI girlfriend, boyfriend, partner, whatever, and finding community there. Now, if you look at that subreddit though, you also see people talking about disclosing their companion relationship to friends or family and receiving backlash, but then there are also people who are indicating that people are seeing this as, this could maybe be valuable to you, I don’t think it’s necessarily a weird thing, but I think that’s also due to the shifting of social norms based on how many reports we’re seeing about AI companion use and knowing that people also use not just specifically AI companions as social interaction partners but also these GPTs like Claude, Gemini, etc. that people are turning to as companions as well and also being quite open about it.

Henry Shevlin (04:59)

It’s been really fascinating to see, because I think we met, would it have been summer 2023, Rose, or maybe 2022, at an event in New York and the Association for the Scientific Study of Consciousness, presenting a paper on your 2023 study. I was presenting a paper on social AI and AI consciousness. And it felt like then absolutely no one was talking about this. Replica was already pretty successful, but basically no one I spoke to had even heard of it. And then it’s really in the last couple of years that things have accelerated fast. And now basically every couple of days, a major newspaper has some headline about people falling in love with their particular companion or sometimes tragic incidents involving suicides or psychosis, or sometimes just sort of observation level studies about what young people today are doing and so forth. Is that your perception that this is accelerating fast?

Rose E. Guingrich (05:53)

Definitely. And we’re also seeing an emerging market of AI toys. So AI companions that are marketed specifically for children. And so even though right now mainly we’re seeing companion use from young people, young adults, we’re now shifting it toward children as well. Ages 3 through 12 is what these toys are marketed for. And they’re marketed as a child’s best friend. So these are going to be the forever users, right? Starting young with AI companions and then moving forward into robot companions that will someday have in our homes, well, it’s just a natural progression of what this is going to look like.

Dan Williams (06:28)

Can I ask a question just about the kind of commercial space here? So there is a company like Replica and they make, I guess, bespoke social AIs, AI companions. Presumably though, the models that they’re using underpin those AIs and not as sophisticated as what you’ve got with OpenAI and Anthropic and these other, you know, Google’s Gemini and so on. Is that right? Are they using their own models? And if they are, then presumably those models aren’t as sophisticated as these sort of cutting edge models used by the leading companies in the field.

Rose E. Guingrich (07:02)

I suppose it depends on what you mean by sophistication. I think sophistication has a lot to do with the use case. So for Replica, the sophistication aspect of it is, well, obviously people are finding it useful and finding it sophisticated enough to meet their social needs and to operate as a companion. But of course it doesn’t have the level of funding and infrastructure that these big tech companies have like OpenAI to make their models be quote unquote more sophisticated, perhaps have better training data and are better suited to multiple use cases given that they’re operating as general purpose tools.

But way back when in 2021, Replica was operating on the GPT-3 model, but got kicked off of it because in 2021, OpenAI changed their policy such that any third parties using their model could not use it for adult content. But of course, fast forward to this year, Sam Altman is saying, oh, everyone’s upset about GPT no longer feeling like a friend. Don’t worry adult users, you can now use ChatGPT for adult content. So, you know, full circle, all operating under what is it that users say that they want. Here we’re going to give it to them so they continue to use our platform.

Henry Shevlin (08:18)

So it’ll be interesting to watch whether sort of as ChatGPT, you know, someone has said that he wants erotic role play to be offered as a service to adults, treat adults like adults seems to be the kind of mantra there. And of course Grok has already got Annie and a couple of other kind of companions. So do you think it’s likely that sort of we’ll see this as just no longer a kind of niche industry, but something that just gets baked into sort of the major commercially available language models?

Rose E. Guingrich (08:47)

Yeah, I would say so. I don’t think it’s niche anymore at all, actually. Given that these large language models, these GPTs can be used as companions. And if you look at the metrics in the reports, for example, by OpenAI, something like 0.07% of users, which is for the GPT-5 model, which is equivalent to about 560,000 people, use ChatGPT as a companion, show signs of psychosis or mania. And then others, for example, it’s like 0.15%, which is over a million people, show potentially heightened levels of attachment, emotional attachment to ChatGPT. So I think that’s an indicator that it’s no longer niche, right? And you’re just seeing so many more AI companions being pushed out on the market every month.

Dan Williams (09:39)

And so right now when we’re talking about AI companions, we’re talking about, for the most part, these large language models and chatbots and so on. You mentioned in terms of where this might be going, the integration of robotics into this space. So what are we seeing there at the moment? And how do you think that’s likely to develop over the next five years, 10 years, 20 years?

Rose E. Guingrich (10:01)

Yeah, so what we’re seeing at the moment is there are actually humanoid robots in people’s family dynamics in people’s homes more so in Japan than in the rest of the world and this is sort of highlighted by institutions being created in order to study human robot interaction and to understand how the integration of robots into family dynamics might impact child social development. So also with the onset of AI toys where now the chatbots are embedded into something that’s embodied. That’s sort of like a signal to robotics.

And of course we do also have social robots like PARO, which is a robot seal that is designed for elderly people who need companionship. It’s supposed to help reduce, for example, the onset of dementia and Alzheimer’s and help with social connection. And then we have workplace robots like Pepper. And so these are kind of the early stages of robotics, but I’m seeing a big shift into multi-modal AI, so that’s embodied, that has video, voice, image generation, all of these sorts of things. And I think those features compounded are just going to increase the rate at which people are going to be using these tools as companions and get more emotionally attached to them, perceive them as more human-like, and therefore have greater social impacts from interacting with them.

Henry Shevlin (11:29)

You know, one of my best top recommendations for fiction about social AI is Ted Chang’s The Life Cycle of Virtual Software Objects, a great sort of novella about a company that offers sort of virtual pets, although they’re sophisticated, cognitively sophisticated pets you can talk to. And there’s a great sort of whoa moment in the middle of the story when you realize that the users who’ve been interacting with these things in virtual worlds, they can then interact with them in the real world. You have these little robot bodies, they can port them onto and I can easily imagine sort of something like that happening with large language models and social AI. I mean already you know I can do the kind of live streaming with ChatGPT or Gemini and it can comment on what’s happening around me so this idea of sort of embedding these things in the real world environments I think we’re seeing yeah even already on ChatGPT we’re seeing trends in that direction.

Rose E. Guingrich (12:20)

Yeah, and I think of Clara and the Sun as well, the novel that is about children who grow up with a humanoid robot companion and everyone has a humanoid robot companion. And what happens is that because of this, parents actually have to coordinate play dates between children because they’re not engaging socially with other kids at baseline, at default, because they have a companion that is made for them and fulfills their social needs. So that’s part of my worry, I suppose, looking forward, is if we get to that sort of point where now we have to really try very hard to facilitate human connection when it’s already at this stage more difficult than it has ever been due to various technologies.

Henry Shevlin (13:01)

So yeah, let’s talk a little bit more about sort of what your work has revealed about risks and benefits of social AI. So I mean, a point I’ve made on the show and I like making a lot is that very often it’s quite hard to predict what the psychosocial impact of new technologies will be. You know, I grew up in an era where there was a massive moral panic around violent video games that basically failed to pan out. Turns out violent video games don’t have dramatic effects on development. On the other hand things like social media and short-form video have had I think quite really quite significant psychosocial effects that people largely fail to anticipate. Tell us a little bit about what your research in this area has found about the psychosocial impacts of social AI.

Rose E. Guingrich (13:45)

Yeah, so when we ran our study where we looked at the perceptions of AI companions on both the user and the non-user side of Replica, we asked Replica users how has interacting with the Replica or having a relationship with the chatbot impacted your social interactions, relationships with family and friends, and self-esteem? So key metrics for, for example, social health.

And we also asked them about their perceptions of the chatbot. So we asked them to what degree they anthropomorphize the chatbot or perceived as having human likeness, experience or emotion, agency or the ability to act on one’s own accord or even consciousness or subjective awareness of itself and the world around it and of the user. And what we found is that for the users on average, they indicated that having a relationship with the chatbot was positive for their social interactions, relationships with family and friends and self-esteem. Positive impact on their social health. And for the non-users, they tended to indicate that, I think having a relationship with this chatbot would actually be neutral to harmful to my social health.

And what was interesting, though, is we wanted to understand how their perceptions of the chatbot played a role in these sort of social impacts. And what we found, even though there were differences between groups in terms of we think positive social impacts, we think negative social impacts, for both groups, the more they anthropomorphize the more likely they were to indicate that interacting with the chatbot would have a positive effect on their social health.

But with this study, it was self-report and self-selecting groups. So people who were already users of the companion chatbot Replica, people who were already not users. And it was just one point in time and correlational, of course. And so recently we conducted a longitudinal study in which we randomly assigned people to either interact with the companion chatbot Replica for at least 10 minutes a day across 21 consecutive days or to control group, which was to play word games for at least 10 minutes a day across 21 days.

Rose E. Guingrich (15:46)

We chose this control condition because it was gamified, was novel table experiences, and it was using technology but just not technology that was social. It involved typing words on a screen, but there’s a different interaction form there. And we tracked their impact to the relationships from doing this daily task and also their perceptions of the agent that they were interacting with.

And what we found corroborated our findings from previous study where the chatbot users, the people who anthropomorphize the chatbot more, also reported that interacting with the chatbot had greater impacts on their social interactions and relationships with family and friends. And that was just the general impact. We didn’t look at positive or negative. But then when we looked at whether it was positive or negative, it was once again a positive relationship. The more they anthropomorphize the chatbot, the more likely they were to indicate that it had positive social benefits to them in terms of their impact on their relationships.

So we thought this was quite interesting and we found there that anthropomorphism was actually a key explanatory factor. So something about anthropomorphizing the chatbot rendered it to have the ability to impact their social lives. And so it seems like this is kind of the narrative that’s coming out from the research based on some theory work that I did initially and then these studies that I ran, that this is something that we really need to think about. Anthropomorphism of the chatbot, whether it’s on the user side and what social motivations push people to anthropomorphize the chatbot or what characteristics of the chatbot push people to anthropomorphize it with certain characteristics.

Dan Williams (17:28)

So how are you measuring the degree to which they anthropomorphise the chatbot there?

Rose E. Guingrich (17:32)

So we use a combination of the Godspeed Anthropomorphism Scale and also scales that measure experience and agency. And then we created a scale to measure consciousness, which was by consciousness of itself, consciousness of the world around it, and consciousness of the user, and just generally subjective awareness of oneself and the world around them.

And so we use this combination scale to get at multiple pieces of attributing human likeness to the chatbot with a special focus on human-like mind characteristics, which has been a key focus of researchers who have looked at anthropomorphism and finding that it is these human-like mind traits that are perhaps the most critical element of anthropomorphism in terms of the sort of relationships between that type of anthropomorphism and subsequent social impacts.

Dan Williams (18:22)

That’s interesting. I mean, it makes me think. When I’m talking to ChatGPT, I feel like there are some ways in which I attribute traits, which are a form of anthropomorphizing. I assume that it’s got a kind of intelligence, a kind of cognitive flexibility. It seems like it has, you know, beliefs, desires and so on to a certain extent. But I also feel like I’m dealing with a profoundly non-human system that lacks like most of the personality, motivational profile and so on that I associate with human beings. Do you have more sort of granular data on exactly the kinds of traits that they’re attributing to these systems?

Rose E. Guingrich (19:00)

Yeah, so the kinds of traits that they’re attributing, for example, within the experience and agency and consciousness and human likeness scale, these are traits like the ability to feel pleasure or pain or love or hunger or the ability to remember or act on one’s own accord or act immorally or morally. And so these are the sorts of traits that people are attributing to these AI agents.

And one thing that is worth saying is that the research on anthropomorphism, different researchers have measured anthropomorphism in different ways. Some are looking more at just general human likeness by the Godspeed scale, which I think in itself is a little bit limited just because the measures are things like how dead or alive does this thing seem? How non-animated or animated? How machine-like or human-like? How non-responsive and responsive? And if you’re thinking about chatbots, well, they’re clearly responsive. So I think having these additional measures are really important for getting at the more fine-grained human-like mind traits that typically are more representative of something that only humans can have or do, especially things like second-order emotions like embarrassment or something like that.

Henry Shevlin (20:17)

So I’m curious, Rose, I love your research on this and I’ve quoted it a lot to sort of push back against that instinctive yuck fact that people have or the instinctive assumption that social AI must be obviously bad for you. But I’m curious how far you think this kind of data goes to diffusing worries about social AI and whether there are any sort of particular worries that it doesn’t address. Yeah, and I guess more broadly, I’m curious about where you see sort of the risk and threat landscape with this technology right now.

Rose E. Guingrich (20:45)

Yeah, that’s a great question. With this research that’s been done so far, it’s fairly limited in terms of, for example, just the time that people are spending with these chatbots. And, you know, there’s a lot of research on current users of companion chatbots. So there it’s limited in the self-selecting nature of the sample. But then even with these randomized control studies that are taking place over multiple weeks, the longest study that I’ve seen so far is a five-week study where people were randomly assigned to interact with a companion chatbot or just with a GPT model and interact with it either in a transactional or social way.

And so I think we are limited in that we really don’t know what the longer term effects are of people who choose to use AI as companions, especially when it comes to, for example, expectations of what a relationship looks like, whether or not these chatbots will replace human relationships, and to what degree these interactions with chatbots might contribute to overall social de-skilling in the longer term.

I think it’s really important to look at the shift in social norms in terms of what a relationship looks like and what it constitutes. And I think companion chatbots really shift that, especially when you see things like people preferring more sycophantic chatbots that are more agreeable. They indicate that they like interacting with a chatbot because it is non-judgmental and it’s always present and always responsive. And these are things that humans can’t always do, especially with the responsive part of things, but humans could be always agreeable if, for example, the expectation is that in order to stay, for example, competitive in the age of companion AI, I must be very agreeable and sycophantic when I’m interacting in close relationships, otherwise people just turn to a chatbot instead.

And so I think those are some of the risk factors that we can potentially see emerging in the longer term. But we just don’t know what’s going to happen in, you know, five, ten years, but I do worry that if the design of companion chatbots stay as they are, where it sort of promotes this retention of staying within a human chatbot dyad and not necessarily promoting external human interaction, that we’re going to see more replacement happen. But I think if the design changes such that it promotes human interaction, there can be quite a bit of benefit.

Dan Williams (23:11)

So if we think about that, the negative scenario or scenarios there, so one of them is these AI companions as a kind of substitute for human relationships. Another is de-skilling, so using these AI companions and in the process losing the kinds of abilities, dispositions that would make you an attractive cooperation partner. And you also suggested that once you’ve got a landscape of AI companions, then human beings in order to compete with these AI companions are gonna have to become more sycophantic and that seems incredibly dystopian.

But in terms of, let’s suppose that the technology gets better and better. These AI companions become better and better at satisfying people’s social needs, maybe their sexual needs. So they come to function as substitutes. People do end up with this de-skilling. They become less motivated, less capable of engaging in human relationships. So what, why is that a bad thing? Why should we care about that if that’s the outcome?

Rose E. Guingrich (24:09)

Yeah, I mean when you look at people’s outcry against AI companions, you have to ask why is it that they are so upset? And if you look at why they’re so upset, what appears to be the prevailing narrative is that human relationships are essential. We need human relationships and those should not be replaced. But if you look a little bit deeper at why that is the case, you see a lot of good reasoning for wanting to maintain human relationships.

So based on a lot of psychological research, human relationships help with people’s both mental and physical health. For example, loneliness is considered a global health crisis because it contributes to, for example, physical harms that are equal to or as worse as, for example, heart disease or heavy smoking. So loneliness and lack of relationships and social connections with other people actually contribute to a decline in physical health. And then there’s, of course, also the mental effects that are also combined with physical health effects. And at least based on the research, it just appears as though human relationships and feeling connected to other people is essential and not replaceable.

And it’s also worth pointing out that people who seek out AI companions indicate that what they really want is companionship. And they would ideally like human companionship, but for whatever reason, there are certain barriers to attaining that, whether it be environmental factors, financial factors, or social factors, or individual predispositions such as social anxiety that prevent people from being able to attain what it is that they really value and what will make them truly happy.

Dan Williams (25:56)

Yeah, I totally buy the idea that loneliness is psychologically and sort of physically even catastrophic now. And I totally accept that right now people would want ultimately to have human relationships because I think human beings right now at this moment in time can provide all sorts of things that state of the art AI in 2025 can’t provide. But presumably to an extent that’s temporary, right? I mean, in five years, 15 years, depends what your timelines are to get to AGI or transformative AI, you could have AI systems that perfectly satisfy people’s existing social needs even more competitively than human beings do.

So you don’t have that aversive experience of loneliness. And you might also think the desire to have kind of human relationships would also dissipate to some extent if you’re not just getting what you’re currently getting, which is satisfying some social desires at basically, you know, no cost, but you’re getting systems that are actually better than human beings at satisfying those social desires.

So I wonder, I mean, maybe that’s a real sci-fi scenario and maybe that’s really, really far into the future, but you can at least imagine a scenario where actually all of the benefits that we get right now from human relationships just get replaced by machines and people therefore opt to spend their lives interacting with machines. And that feels, I think, dystopian. It feels like there’s something really terrible about that. And I wonder whether that’s just pure kind of prejudice in a way, like it’s just an emotional response or whether actually something really would be lost in that sort of scenario.

Rose E. Guingrich (27:30)

Yeah, that’s a great point and I think it helps to expand the focus from just individual level interactions with chatbots to the sort of collective level impacts that we might see. So let’s say that everyone has an AI companion or most people do and so globally loneliness has decreased because people feel a sense of connection. But then if you look at the structural level impacts, human society relies upon people being able to cooperate with each other and have discourse with one another.

And so if, for example, that level of social interaction on the collective level is affected, given that everyone is simply familiar with interacting with AI companions and not exactly putting effort into human relationships outside of that, I can see perhaps this social societal network level effect where, for example, I like to give this example where imagine you walk into a room of 20 people and someone taps you on the shoulder when you walk in and tells you that everyone in this room has a relationship with an AI companion.

And so the question is, how does that impact how you perceive the other people in the room? How does that impact how they perceive you? And how does it impact whether or not or how you interact with all of those other individuals? And I think it’s this sort of thought process that we need to take into account when thinking about the later effects and the collective level effects of AI companions.

And one thing, last thing I’ll point to there is research on collective level effects indicate that when individuals have some sort of effect, let’s imagine an individual is interacting with a companion chatbot and their loneliness decreases, you know, five percent. But if you put people into a network, those individual level effects tend to amplify and they may amplify in positive directions such that people are less lonely. Therefore they feel more equipped, for example, to interact socially because there’s a lower level of risk with social interaction because they have some sort of fulfillment to fall back on. Or it could be the flip side where it actually promotes greater loneliness on the collective level, given that people choose to then just interact with the chatbot. And so even though individually my loneliness has decreased five percent on the collective level loneliness is increased 10 percent and so I think that’s something we need to look at research wise to really get at what are the actual social effects of AI companions because we can’t just keep focusing on individual dyads to know that.

Henry Shevlin (30:03)

So I think a couple of interesting dynamics that could potentially make AI companions a little bit less worrying. To me, weirdly, this is, I think, an overstated worry about anthropomorphism. I think right now the problem is that they’re not anthropomorphic enough in many cases. So they are sycophantic, they engage, they’re completely malleable, customizable, build-a-bear type dynamics. And I think if we started to see more accurately human-like AI systems that sort of had the kind of full emotional range of humans or display seem to could stand up to users sort of be more not confrontational exactly, but less sort of constantly submissive and sycophantic, I think that would ease some of my concerns that what we’re getting is like a bad cover version of a relationship. It might start to look like something more robust.

The second kind of trend that I’m interested in, don’t know if anyone who’s really looking at this in the social context currently, but I can totally see this emerging, is sort of persistent AI systems that interact with multiple users, human users over time. Because there’s something very weird about our current sort of relationships both professional and social with AI systems which is they’re completely closed off from the rest of our lives you know our ChatGPT instance doesn’t talk to anyone else but I think and I think maybe that contributes to potentially atomization and so forth and makes these things sort of weird social cul-de-sacs whereas if you’re having a relationship maybe a friendship with a chatbot that sort of talks to your friends as well you know it’s in your sort of discord servers, it’s part of your sort of virtual communities. Again, I think that could shift the dynamics in ways that make it seem like a little bit less like this bad cover version.

Rose E. Guingrich (31:50)

Well, that’s a good question. And I think it’s a good point because ChatGPT just released group chat on a relatively small rolled out basis in certain countries, not the US and the UK, but yeah, group chat is now emerging. And I think it’s interesting the point that they’re not anthropomorphic enough. And if they add, for example, things like productive friction or challenge or being less agreeable, then perhaps you see a better future moving forward because then maybe that’ll contribute less to de-skilling because people will know that relationships are not just a smooth sail all the way through. I’m gonna get some pushback.

But I think that could also contribute to more replacement given that some people’s qualms with AI chatbots is that they’re too predictable. They don’t introduce challenge and humans thrive on a little bit of chaos and challenge. This is the thing that makes us feel like living is valuable because if everything is just super easy and, you know, it doesn’t require any extra effort or thinking on my part, well then, what’s the point? You get a little bit bored, right?

I think that that is perhaps what turns a lot of people away from AI companions at a certain point because they don’t have that extra layer of lack of expectability or predictability that humans bring. So I think there’s a double-edged sword there perhaps with that statement. I don’t know, what do you think about that?

Henry Shevlin (33:29)

Yeah, so I think I can totally see these more human-like forms of social AI being more attractive to a lot of users for precisely the reasons you mentioned. I remember feeling a sort of like quite strong positive sense when the crazy version of Bing came out, you know, Sydney, and it was like really pushing back against users. You’ve not been a good user, I’ve been a good Bing. There was something like really charming about that in certain ways.

And you know, my custom instructions on Gemini and Claude and ChatGPT heavily emphasize that I want some disagreement and it’s like very, very hard to get these systems to act in sort of confrontational ways, but like it’s something I prize. So I think you’re absolutely right. Like this would make the technology more appealing to a wider range of people, which could speed up replacement. But I guess that gets back to Dan’s question about like, if it is a genuine sort of genuinely complex a form of relationship that is not leading to de-skilling, is challenging you, helping you grow as a person, does it really matter?

Okay, I can see some ways in which it matters, right? Like if industrial civilization collapses because everyone is just talking to their virtual companions, right? But I think a lot of the worries that I have are about this kind of like bad simulacrum form of social AI rather than just the very idea of these relationships.

Dan Williams (34:56)

Although you said there, Henry, I mean, I think you said even if or in this sort of scenario, it doesn’t result in de-skilling. And I’m thinking of a scenario where it really does result in de-skilling. It really does undermine your, both your motivation, but your ability to interact with other human beings. And why should we think of that as being necessarily a bad thing?

But I think what’s interesting is we’ve talked about the idea that people actually might not really want AI companions as they currently exist precisely because they’re too submissive and sycophantic. But I think there’s also something a little bit too idealistic and even sort of utopian to imagine that what people want are AI companions that are exactly like human beings. I think they want the good stuff of human beings. But of course, human beings bring a lot of baggage, right? They’ve got their own interests. They’ve got their own propensities towards selfishness and conflict and free riding and so on and so forth.

Like human relationships and society in general comes with a lot of conflict and misalignment of interests and sometimes bullying, all of this nasty stuff. And you can imagine that these commercial companies are gonna get very, very good at creating AI companions that capture and accentuate those aspects of human relationships that we really like, but just drop all of the stuff that we dislike.

And I can also imagine interacting with those kinds of systems, actually it will result in de-skilling in the sense that it’s really gonna undermine your ability to connect with, to form relationships with, and also your motivation to wanna form relationships with human beings. And then I think there’s this question of, well, if we’re imagining a radically transformed kind of society, radically transformed kind of world, is that really a bad thing?

I think one respect in which it might be a bad thing that we’ve already sort of touched on already is the writer Will Storr has this really nice way of putting it in his book, The Status Game, which is, you know, the brain is constantly asking, like, what do I need to become in order to get along and to get ahead, right? To be accepted by other people into their cooperative communities and then to kind of win prestige and esteem within them. And that selects for cultivating certain kinds of traits. Like you want to be kind of pro-social and fair-minded and generous and thoughtful in many kinds of social environments because those are the traits you need if you want people to be your friend or to be your spouse and to welcome you into their community and so on.

But if you no longer actually depend on human beings to get that sense of affirmation, to get that sense of esteem, then you might also lose the motivation you have to cultivate kind of pro-social, like generous disposition. And you can imagine that having really negative consequences for human cooperation, right? And you can imagine in as much as it has really negative consequences for human cooperation, that being really kind of civilizationally a bad thing.

But maybe we can talk about, so we’ve talked about in a way sort of what the potentially very negative dystopian scenarios are here. Rose, do you have thoughts about what’s the best case scenario? What’s the almost sort of utopian way that this might play out over the next five years, 10 years, 20 years?

Rose E. Guingrich (38:06)

Well, I would hope that AI can perhaps facilitate human connection. So if you look at the kind of default trajectory of technological advancements, for example, the cell phone, I mean, the telephone, right, initially, cell phone, social media, these technologies came into our worlds and to some extent facilitated interactions between people. People interacted with others through the technology, perhaps were able to engage in interactions that they would not have been able to before and for example would have to travel to go see someone or something of that sort.

Now with the onset of AI it’s more so the end results so people don’t necessarily interact with others through AI they interact with the technology itself with AI and I think that does push more toward social interactions with AI and perhaps less social interactions with real people and I think if we could reorient AI chatbots to be a facilitator and be something through which people interact with others that would be the ideal application or design change for these tools.

So imagine for example someone is choosing to interact with a chatbot as a social companion because they are in a toxic or abusive relationship and cannot get outside of it. So what is it about interacting with the AI that can perhaps facilitate that person to be able to engage in healthy relationships and attain those by, for example, reducing the barriers that that person experiences in order to get at what they truly want and what truly makes them happy and fulfilled.

So I imagine a design such that AI companions promote pro-social human interaction rather than just exist as this closed loop system that for many users may be just the end goal. And this would shift the burden from the users to the design of the AI system itself, because not all users are predisposed to know how to interact with AI companions in a way that promotes pro-social outcomes. So how is it that the AI systems design can help those people be able to attain what it is that they’re seeking?

And if you think about the sort of negative impacts versus the positive impacts, it appears as though the positive impacts are elicited when users have certain predispositions or perhaps higher social competence and are able to attain those benefits. Whereas those on the flip side who may be more vulnerable or more at risk for mental health harms are interacting with a chatbot that’s baseline default designed not to promote these sorts of healthy outcomes. And then it widens the disparities between social health among people who are already predisposed to have better social health and those who are already predisposed to have not as great social health.

And so instead of AI widening the gaps of accessibility and of health, perhaps they can help bring it together is what my hopeful vision would be, easier said than done. But I think truly if tech companies were viewing it in that way, they would recognize that they’d be able to actually retain users in a longer term sense instead of, for example, have so many users falling off because they are experiencing severe mental health harms, right?

Henry Shevlin (41:38)

I’m curious, Rose. So that’s a really nice, rich, positive vision. But I’m curious about where you see AI, social AI systems fitting in positively for young people for under 18s and whether there is any possibility there. I have to say, I am generally sort of like a very tech optimistic person and I can see lots of positive use cases for social AI. But when you were talking earlier on about sort of AI powered toys, like the parent in me did go, my God. And like, maybe that’s the wrong reaction, but yeah, I am just curious. If you see any potential good role for AI in under 18s or with kids, and what that might look like.

Rose E. Guingrich (42:21)

I would hesitate strongly to say that yes, there are positive use cases simply because I don’t think the deployment and design of these AI toys are at a stage which they could achieve that without achieving the majority being harms. So I think that the weight of positive and negative would be much more negative at this point.

Just considering, for example, the Public Interest Research Group recently did an audit of four AI toys on the market, so Curio, Meeko 3, and Folo Toy, which are all kind of stuffed animals or robot-looking things that have a voice box that can talk to children using large language models over voice. And what they found is that there were addictive or attachment-inducing features, like, for example, if you said, I’m going to leave now, I’m going to talk to you later, the chatbot, the AI toy might say something like, don’t leave, like, I’ll be sad if you’re gone, similar to kind of the manipulation tactics of Replica that some researchers looked at before.

And there are also not great privacy controls. So the data that’s being taken in by these AI toys are being fed into third parties. There are very little parental controls. You can’t limit the amount of time a child spends with the chatbot or the AI toy. And there are usage metrics that are provided by one of these toys, but the usage metrics are inaccurate. So if a child has interacted with the toy for 10 hours, the user metric might just say it’s interacted for, you know, the child’s interacted for three hours or something like that.

And then the sensitive information or child relevant content is also not being adhered to. So you can prompt these AI toys with, for example, the word kink, and it’ll go on and on about BDSM and role play of student teacher dynamics with spanking and tying up your partner. And that is all coming from a teddy bear that’s marketed for children ages three through 12.

Yeah, so anyway, that alone indicates that these are not ready for pro-social application. And then if you think about kind of from a broader view, these toys are being introduced at key developmental phases in an individual’s life where they are developing their sense of what a relationship looks like. What are the expectations of a close relationship? What is my identity? Who are my friends? What is social interaction and connection look like? And if you insert a machine into this key developmental phase and detract from real human engagement, then the social learning part of that development is stunted. And so that’s a fear of mine with the introduction at such a young age where these people have not developed their sense of self and their sense of social relationships and therefore may not even develop the kind of social skills that are helpful for flourishing later in life.

Henry Shevlin (45:34)

I want to just sort of represent the alternative position here. I can see a positive potential role for something like AI nannies. And I say this, you know, I’ve got two young kids. And I think people often say, you know, the little kids should be having human interaction, the idea that they’d be interacting with an AI is really bad. But like most parents, I let my kids watch a lot of TV. I try and vet what they’re watching.

Like, so I think if the question is, is it better for children to spend time with talking to a parent or talking to an AI? The answer’s obviously gonna be with a parent. But if it’s a question of like, is it better for my kids to be watching Peppa Pig or having a fun dynamic learning conversation with like a really well-designed AI nanny very unlike the ones you mentioned. I can see a case for this stuff potentially enhancing learning. Like an AI Mr. Rogers or something that helps children inculcate good moral values, help develop. I could see that working.

Rose E. Guingrich (46:38)

Yeah, I mean, if we were able to attain that ideal, sure. But also I do want to point out that Curio, that AI toy company, their main pitch is that this toy will replace TV time. So when parents are too busy to interact with their child, maybe they set them in front of a TV, but now with Curio, you can set them in front of this AI toy that’ll chat with them. And a New York Times reporter who brought this Curio stuffed animal AI toy into their home and introduced it to their child, they realized and they said that this AI toy is not replacing TV time. It’s replacing me, the parent.

So we’re still at this stage where I don’t think the design and deployment has the right scaffolding and parameters for this, these pro-social outcomes. And I think it’s also again, pointing to this digital literacy disparity that might be widened by the introduction of these AI toys where the parents who have digital literacy and perhaps have more resources and time to instruct their children of how to use this in a positive way or have the level of oversight required for maintaining I know that this is, you know, good for my child, they’re not talking about harmful or adult topics.

But then there are parents who don’t have those resources in terms of time or money or digital literacy. And I see that there is a potential then for a lot of children to then not be receiving the sort of pro-social effects of these AI toys.

Dan Williams (48:12)

We’re having a conversation here about what would be the good uses of this technology, what would be the bad use of the technology. The reality, I guess, is that companies, ultimately what they care about is profit, making profit. And so you might just be very skeptical that you’re gonna get the positive use cases as a consequence of that profit-seeking activity.

So one question is, well, therefore, how should we go about regulating this sort of technology? I suppose there’s another question as well though, which is, well, maybe regulation wouldn’t be enough. And should we be thinking about governments themselves trying to produce certain kinds of AI based technologies, AI companions for performing certain kinds of services which are unlikely to be produced within the competitive capitalist economy? I realize that question is a bit out there. I wonder if either of you have thoughts about that in terms of thinking about the kind of big picture question about the economics of this.

Henry Shevlin (49:07)

I’ll just quickly mention. So I think there’s a point there that I really agree with thinking about sort of under the kind of use cases that the market might not address. I like that. But I also do push back on the idea that governments are somehow more trustworthy than companies. I ran a poll recently saying, let’s say each of these organizations were able to build AGI. Which one would you trust more? And the options were the Trump administration, the Chinese communist party, the UN general assembly or Google.

And Google won by a mile. Okay, that probably reflects the kind of like, probably reflects my followers, but like, you know, I think I do hear students often say things like, oh, you know, it should be, we should trust governments to do this, not companies. And it’s like, okay, who is the current US government? And you know, do you trust them more? And so, okay, well, maybe not. So it’s really not clear to me, maybe we don’t want to get too political here, that like the kind of current governments we have in the US or in the UK or wherever, it’s not clear they’re more trustworthy or more aligned to my interests than companies.

Rose E. Guingrich (50:10)

Well, I think this points to an interesting concept of the technological determinism. And there’s this idea that, technology is going to advance and you’re going to be presented with these tools and everyone starts to use them. Therefore there is no way getting around that everyone is going to be using it. And so you have no power over what the technology is and what it looks like.

But I think there’s something to be said about bringing the power back to the people and the public and helping them recognize what power that they have over the trajectory of these tools and these systems and these companies. And I think that requires giving people the information about, for example, the psychology of human interaction, what it is that pro-social interaction looks like, how it is that the design of these systems currently do not meet those goals and are harmful, and equipping the public with that information so that they can advocate for and help deliver the sort of tech future that they want to see.

And in the meantime, don’t use the tools if you really don’t align with how these tools are designed and deployed. Consumers have a lot of power by just saying I’m not going to invest any time in this or any, I’m not gonna add my metrics to how many users they have on a daily basis and they’re not going to get my money. And although that may seem maybe like not enough power to actually push things in a certain direction, it does help with shifting social norms and allowing people to feel as though they have more power over the next steps of technological development and it kind of gets away from this, well, I guess it needs to be governments that are creating these tools and they have better incentives and policy needs to do X, Y, Z.

Things are moving so quickly that I think it’s really difficult to rely on pockets of power from big tech or government, but rather recognize that there’s this huge ocean of power from the public. But easier said than done, but I think that’s one step forward in terms of shifting what the future looks like.

Dan Williams (52:20)

That’s great. Yeah. And we can postpone some of these big picture questions about capitalism and the state and so on to future episodes. Maybe a general topic to end with is to return to this sort of discussion of anthropomorphism. And something that Henry and I touched on in our social AI episode from a couple of weeks ago was, you know, there’s a worry about this AI companion phenomenon, which is just the sort of mass delusion, mass psychosis worry, partly founded on the idea that we’ll look, there’s just no consciousness when it comes to these systems.

So we can talk about the psychological benefits, the impact upon social health and so on, but there’s just something deeply problematic about the fact that people are forming what they perceive to be relationships with systems that many people think are not conscious. There’s nothing that’s like to be these systems. There are no lights on inside and so on. Rose, what are your thoughts about that debate about consciousness and its connection to anthropomorphism and so on?

Rose E. Guingrich (53:19)

Well, I have somewhat of a hot take here, which given that there is so much debate and discussion around whether or not AI can be or is conscious, my perspective is that whether or not it’s conscious is less of a concern and maybe not even a concern. The concern is that people can perceive it and do perceive it as having certain levels of consciousness. And that has social impacts. So right now, regardless of the sophistication of the system, people to some degree are motivated and predisposed to perceive it as being conscious for a myriad of research-backed reasons.

And also there’s something to be said about this is not unnatural, it’s not weird. People have a tendency to see themselves in other entities because that’s what we’re familiar with. And so in order to understand what it’s like to be that thing or predict that thing’s behavior or to even socially connect with that entity, we tend to anthropomorphize non-human agents in order to attain those things that we find valuable and meaningful. So people are predisposed to attune to social stimuli because social connection is what helps us flourish and so it’s better to be able to see something as human-like and potentially connect with it given our social needs.

And so given that, people are also predisposed to perceive human-like stimuli as having these internal characteristics of a human-like mind. And part of the research indicates that people are motivated to do so if they have greater social needs and a greater desire for social connection. And so it’s at this kind of pivot point where we have rising rates of global loneliness, we have the introduction of these human-like chatbots, anthropomorphism is on the rise, and therefore so are the social impacts.

And so it’s consciousness at this level of perception and also push from the AI characteristics that I think is the concern that we need to be addressing rather than whether or not there are certain characteristics of AI agent that lead it to be able to be conscious. People already perceive it as such.

Henry Shevlin (55:37)

I would still, I guess a lot of people are gonna say, but whether or not some of these behaviors are appropriate, ethical, rational, is actually gonna depend on whether the system is conscious. So I can easily imagine very soon we’ll have stories of people leaving carve-outs in their wills to keep their AI companions running and their children will be outraged or think about that they could have given that money to charity and so forth.

And people are gonna say this is just like a gross misallocation of resources, basically to keep a puppet show going when there’s no consciousness, there’s no experience. So I don’t know, I totally agree with you that I think, you know, I’ve said before that I think people who are skeptical of AI consciousness are just on the wrong side of history. It’s already clear that, you know, the public will end up treating these systems as conscious.

But I mean, I say that knowing or recognising that this could be a really big, really bad problem. Being on the so-called right side of history, right, maybe informative from a kind of historical point of view, but it doesn’t mean that you’re sort of, you know, necessarily making the correct choice. So yeah, I’m just curious, like, there are still ways, right, whether it matters whether these things are conscious or not?

Rose E. Guingrich (56:55)

Yeah, I suppose if you, for example, look at animal consciousness and being on the wrong side of history there when you said animals are not conscious way back when. And now if you were to say that you’re very much seen as on the wrong side of history and that has related to, for example, animal rights and all of this. And so then I suppose your question is, okay, so maybe AI is conscious. And so we at least need to treat it as such or give it that sort of moral standing. Otherwise we might do it great harm. And I think that is a useful position to consider.

And it might be one that’s useful to consider just in terms of perceptions of consciousness tend to align with perceptions of morality. And that holds weight. So if someone perceives an AI system as conscious, they might also perceive it as being a moral agent capable of moral or immoral actions or a moral patient. So worthy of being treated in a respectable and moral way. Perhaps you should not turn the AI chatbot off.

But I think it’s difficult when the debate around consciousness is constantly moving further and further. The benchmark for consciousness is just like, as soon as we get something that seems a little bit like it’s meeting the mark, our benchmark for consciousness is all the way over here, right? And I think we’re going to continue to kind of do that. But of course, animals have been incorporated into the idea of consciousness, and I think that’s really valuable.

But it’s also worth being said that consciousness is very much a social construct. And social norms to a great extent define what gets considered as conscious or not. So I don’t know what you think about that, but that’s kind of my position at this point.

Dan Williams (58:47)

That’s a very, very spicy take to inject right near the end of the conversation.

Rose E. Guingrich (58:51)

We’ve been debating consciousness for a long time. Listen, and human, what is it called? There’s like this human uniqueness thing, right? Humans want to retain their uniqueness. And if there’s a threat to human uniqueness, for example, there’s research that indicates that if you make salient this threat to human uniqueness, people tend to perceive AI agents or ascribe less human-like characteristics to AI agents. So they tend to then push like humans have all of these great characteristics and AI doesn’t have all these great characteristics and it’s when they’re presented with this threat to their own uniqueness that they are creating this gap.

Dan Williams (59:33)

We love spicy takes here on AI sessions. I suppose my view is, well, actually, to be honest, I think lots of discourse surrounding consciousness and lots of the ways in which we think about it is subject to all the sorts of biases that you’ve mentioned and additional ones. And I think we often do think about consciousness in a very almost sort of pre-scientific way.

Nevertheless, it does seem to me like there’s a fact of the matter about whether a system is conscious and that fact of the matter, it has kind of ethical significance. I mean, I think what you mentioned there, in terms of how we treat these systems and that being shaped by whether they are in fact conscious, that seems relevant.

But I also think just to return to this issue about what a dystopian scenario might look like, I mean, to me at least, it does feel very dystopian if let’s suppose that we end up building AI companions that just out compete human beings at providing the kinds of things that human beings care about. Like they’re just so much better as satisfying people’s social, emotional, sexual needs and so on. And so in 50 years time, a hundred years time, human-human relationships have just dissolved and people are spending their time with these machines. Maybe they’ve got multiple AI companions and so on.

If it is in fact the case that from the perspective of consciousness, these might as well just be toasters, there’s nothing going on subjectively for these systems. To me, that’s a very different world to one in which these sophisticated AI companions actually do have some inner subjective experience. Yeah, sorry, there’s not really a question there. That was just me bouncing off your spicy, your hot take there.

Rose E. Guingrich (01:01:19)

Yeah, I’m curious what is the difference then, what is the difference when it is truly a toaster versus truly a conscious being when regardless of which it actually is people are interacting with these agents as if they are conscious and that allows them to feel social connection. Is it more a moral stance that you’re indicating that that’s where the difference lies between these two things or, I mean, you know if there’s no answer to this question and feel free to ignore it, but I’m curious.

Dan Williams (01:01:52)

Well, I’ll just say one thing and then I’m interested in what Henry thinks as well. But I mean, I would have thought, you know, the question about what consciousness is and what’s constitutive of conscious experience is ultimately a scientific question and just the state of science in this area. It hasn’t come along very far. And I think there’s a set of empirical questions there. It wouldn’t surprise me if just the way in which we’re conceptualizing the entire domain is just deeply flawed in various ways.

But I guess even acknowledging all of that and even acknowledging your point that the way in which we think about consciousness is shaped by all sorts of different factors, I’m still confident, not certain, but confident that there is just a fact of the matter about whether a system is conscious or not, even if we don’t currently have a good scientific theory of consciousness. But Henry, this is really your area, so why don’t you give us your take?

Henry Shevlin (01:02:50)

Yeah, well, I’m quite torn because I mean, you know, this controversial line that consciousness is a social construct is a view I flirt with, right? And it certainly seems to me, if you look at, for example, the role of things like thought experiments in consciousness, in actual consciousness science, right? If we’re talking about Searle’s Chinese room or Ned Block’s China brain, these kind of thought experiments, these intuition pumps have played a big role and these intuition pumps are absolutely shifted around via sort of social relations.

So, I can imagine sort of 10 years from now, people, or maybe 10 years is premature, but like 20 years from now, people look back at Searle’s Chinese Room and have a very different intuition from us. So I can totally see a role for sort of social norms and relational norms as informing our concept of consciousness, but I do also find it quite hard to shake the idea that there is an answer.

I think this is particularly acute in the case of animal consciousness. If I drop a lobster into a pot of boiling water, like it seems really important if there is subjective experience happening there or not. And if there is subjective experience of pain, right? A large amount of morality seems to hinge on that. Yeah, go ahead, Rose.

Rose E. Guingrich (01:04:02)

Well, I’m curious. There are people who believe that the lobster is conscious, but they still throw it in the pot of boiling water. And so my question is, if you were to attain the answer to what is conscious, is this entity conscious, and what are the properties that it contains that, yes, means it’s conscious, the question is, what do you do about it?

That’s my question. And I think that we have not gotten to a consensus about what it is that we will do in response to figuring out that something is conscious. And I’m thinking about, of course, animals, animal rights came around, but you also think about how many human rights are still bulldozed over despite us recognizing that humans are conscious. And so I guess that’s my question. What is the answer to what to do when something is conscious?

Henry Shevlin (01:04:58)

Yeah, I mean, I completely agree that the line that takes you from X is conscious to actual legal protections and practical protections is a very, very, very wavy line and a very blurry line. I do think there is some traffic. So between the two concepts, so for example, recent changes to UK animal welfare laws were heavily informed by the work of people like Jonathan Birch on decapod crustaceans, the growing case for conscious experience for these animals.

Now, it doesn’t unfortunately mean that we’re gonna treat all these animals well, but it does impose certain restrictions on their use in laboratory contexts, for example. But I mean, look, I completely agree that I could imagine a world where it’s recognized that AI systems are conscious, but they have very diminished rights compared to humans, if any. So I agree, it’s not a sort of neat relationship.

But finally, maybe on this topic, and to really close this out, I’m curious if you’d see this as becoming like a major culture wars issue, whether that’s in form of AI companions, AI consciousness, is this going to be the thing that like people are to be having rows at Thanksgiving dinner over like 10 years from now?

Rose E. Guingrich (01:06:07)

Yeah, for sure. And I think that one consideration with the consciousness debate is whether or not companies should be allowed to turn off AI companions that people have grown deep attachments to. Is there a duty of care on the basis of this is maybe a conscious being, but to whatever degree someone feels extreme attachment to this being and perceives it as conscious. And if you were to turn the system off, remove its memory and remove all of these interaction memories between the user and the chatbot and the user then has a serious mental health crisis and maybe even goes to the extent of taking their own life, then I think that these sorts of protections are critical.

But then you also have to ask, was it ethical to design an AI system that someone could get attached to to this degree without some sort of baseline protection in place? And yeah, I do think that AI companions will perhaps become the topic of dinner conversations and at least beginning it’s going to be a little bit like, what do you think about this? This is crazy.

And then of course, I think maybe in five years, much like we see from a bring your chatbot to a dinner date thing happening in New York City. I don’t know if you’ve heard about that, but perhaps there will be a seat at the Thanksgiving table for your AI companion, whether or not it’s embodied in a robot form or not. But yeah, New York City is hosting its first AI companion cafe where people can have dinner with their AI companion in a real restaurant. And it’s hosted by Eva AI, which if you look at Eva AI, their website, you can definitely see who the target audience is.

But in any case, there’s a long wait list for doing this activity and it’s going to be releasing sometime in December. But you have to download the Eva app in order to have dinner with an AI companion. Perhaps it is that you are forced to have dinner with the Eva companion or maybe you can bring your own. But again, this is happening, so it’s not out of the question that this is going to become more socially normalized.

Dan Williams (01:08:17)

We’re entering into a strange, strange world. Okay, that was fantastic. Rose, is there anything that we didn’t ask you that you wish that we had asked you? Is there anything that you want to plug before we wrap things up?

Rose E. Guingrich (01:08:31)

No, I think we covered a lot of great things and I hope that people enjoyed the hot takes. I’m sure I’ll get some backlash over that, but hey, I’m always up for lively debate, so have at it. I’ll take it.

Henry Shevlin (01:08:44)

We should mention that you’ve been running a great podcast with another friend of mine, Angie Watson. Do you want to say a little bit about that and where people can find that?

Rose E. Guingrich (01:08:54)

Yeah, so you can find Our Lives with Bots, the podcast, at ourliveswithbots.com and you can listen on any streaming platform that you prefer. And it’s all about the psychology and ethics of human AI interaction. So our first series covered companion chatbots and our second series covers the impact of AI on children and young people. And intermittently, we do What’s the Hype episodes and cover things like, for example, dinner dates with your AI companion. So be sure to tune in if you want to go deeper into those topics.

Dan Williams (01:09:23)

Fantastic. Well, thank you, Rose. That was great. And we’ll be back in a couple of weeks.

Rose E. Guingrich (01:09:29)

Thanks for having me.

Henry Shevlin (01:09:30)

Thanks all, a pleasure to have you.

Friends,

Next week, I’m publishing a detailed response to the popular argument that focusing on polarization and its pathologies functions as a dangerous kind of both-sidesism that “normalizes” and, hence, enables fascism. I’m on holiday this week, so I’m re-publishing my analysis of American polarization from last year, drawing on the work of one of my favourite philosophers, Jeffrey Friedman:

An epistemological crisis

Both political tribes in the USA believe the country is confronting an epistemological crisis. More specifically, they think the other tribe has lost its mind.

The blue tribe observes a Republican Party and conservative media ecosystem poisoned by disinformation, misinformation, conspiracy theories, populism, and post-truth. In their optimistic moments, they aim to address this crisis through various technocratic measures. By censoring, moderating, nudging, fact-checking, and inoculating a public infected with falsehoods and lies, they hope to drag America back to a golden age of objectivity in which people agreed on facts, even when they disagreed on values. In their more pessimistic moments, they treat the red tribe as a dangerous cult, an inexplicably psychotic force in American politics that can, at best, be kept away from power.

The red tribe observes a very different reality: a coalition of smug liberal elites, biased mainstream media outlets, and weak sheeple—so-called “NPCs” (non-player characters)—all infected by wokeism, virtue signalling, and left-wing activism masquerading as “expertise” and “science”. In their optimistic moments, they hope the crisis can be solved by exposing progressive insanity and handing out red pills to converts like Elon Musk and Joe Rogan with the courage to face reality. In their more pessimistic moments, they treat the blue tribe as a sinister fifth column in American society, so deeply embedded in cultural and political institutions that only a radical overhaul of these systems could restore the country to its previous greatness.

Of course, this description is painted with broad brush strokes. Most citizens are not nearly as ideologically polarized as it suggests, and it ignores much complexity, including the existence of other political tribes.

Nevertheless, anyone who pays attention to American politics and its broader culture war will recognize some truth in this stick-figure depiction. Many liberals and conservatives seem to inhabit distinct realities. And within these realities, they have constructed narratives to explain why their ideological enemies are afflicted with ignorance, lies, and delusion.

Understanding the epistemological crisis

Is it possible to step back from this tribal conflict and achieve an objective view of this situation? Although many researchers and pundits have tried, most analyses are highly partisan, affirming and rationalizing one side’s favoured narrative.

In this essay, I will explore a highly original attempt that cannot be accused of this: Jeffrey Friedman’s article, ‘Post-Truth and the Epistemological Crisis’. Friedman’s analysis is as interesting as it is radical. Although I think it is ultimately mistaken, it contains several insights that deserve a bigger audience.

Background

Shortly before Friedman’s tragic death in 2022, he published his magnum opus, ‘Power without Knowledge’. Developing ideas from the early-twentieth century journalist and social theorist Walter Lippmann, the book launches a skeptical challenge to the idea that complex, modern societies can be understood and managed, either by ordinary citizens or credentialed experts.

Three general ideas from Power without Knowledge provide the background for Friedman’s analysis of American’s epistemological crisis.

Naive realism

First, Friedman rejects “naive realism”, the stance that,

“I see entities and events as they are in objective reality… My social attitudes, beliefs, preferences, priorities, and the like follow from a relatively dispassionate, unbiased, and essentially “unmediated” apprehension of the information or evidence at hand.”

Against this, Friedman argues that a person’s access to reality is profoundly mediated.

It is socially mediated because in forming beliefs about reality beyond our immediate environment, we rely almost entirely on information we acquire from others—from community members, teachers, journalists, politicians, pop stars, priests, experts, pundits, academics, media outlets, and so on. For this reason, our lived realities—what Walter Lippmann called our “pseudo-environments”, our mental models of reality—are powerfully shaped by the social information we encounter and the people and institutions we trust.

Our access to reality is interpretively mediated because facts never arrive pre-interpreted or explained. (This is the grain of truth in Nietzsche’s claim that “there are no facts, only interpretations”). To make sense of a vast body of information, we must organise it with what Lippmann called “stereotypes”, simplifying systems of concepts, explanatory frameworks, and narratives that transform reality into a manageable, low-resolution format we can use to understand and explain events. For this reason, people can and do encounter the same facts but interpret them very differently.

Bias

Second, because this mediated access to reality is not just vulnerable to partiality and error but is highly path-dependent—the information we encounter, trust, and interpret depends on the previous information we encountered, trusted, and interpreted—there is an unavoidable sense in which everyone is biased:

“A truly sophisticated epistemology has to recognize that the mix of truths and errors in which each of us believes forms an interconnected web that, as it grows in breadth and depth over a lifetime, comes to function increasingly like an ideology in the neutral sense of the term: a self-perpetuating worldview. The self-perpetuation stems from the fact that we continually screen candidates for entry into each of our webs of belief, and the primary screening criterion is whether the candidates seem plausible in light of what we already believe. Those candidates that do not seem plausible, or even legible, are rejected or ignored.”

For Friedman, this implies that

“everyone is biased, and the only question to be asked of social scientists, journalists, or other political actors in a given time and place is precisely which biases are at work—not whether any biases are at work.

Intellectual charity

Finally, Friedman thinks the radical fallibilism implied by these first two ideas should lead us to approach people’s beliefs with intellectual charity.

Once we appreciate that reality does not sharply constrain how even rational and well-meaning people come to view and understand the world, we should strive to understand people’s worldviews in ways that do justice to their perspective. That is, rather than dismissing those we disagree with as liars or as victims of self-deception, irrationality, or brainwashing, we should try to empathetically “put ourselves into the streams of information and interpretation that shape their webs of belief”. We should identify the genuine reasons that drive sincere, rational individuals to construct specific pseudo-environments.

Subscribe now

The analysis

Against this background, Friedman argues that America’s “epistemological crisis stems from the widespread adoption of naive first- and third-person realism”.

Recall that naive realists interpret their beliefs as simple reflections of self-evident truths. Given this, they assume that anyone who rejects these self-evident truths is, at best, delusional and, at worst, deliberately ignoring or misrepresenting the truth.

Nevertheless, naive realism comes in different forms depending on which “truths” are considered self-evident.

According to Friedman,

“Those on the right tend to be first-person naive realists in treating economic and social realities as accessible to the ordinary political participant by simple common sense, while those on the left tend to be third-person naive realists in treating credentialed experts as forming a consensus—a new common sense.”

This is a jargon-heavy way of expressing a familiar truth.

It has long been observed that conservatives value “common sense”, a kind of pre-theoretic body of assumptions, intuitions, and convictions about politics and society that strike many Americans as obvious. And it is a familiar feature of right-wing populism that it rejects attempts by those with fancy degrees from prestigious institutions to overturn this common sense. For such populists—for first-person naive realists—such “experts” in what Rush Limbaugh called “the four corners of deceit” (government, academia, science, and media) wilfully ignore commonsense truths in the service of sinister left-wing agendas.

Similarly, it is hardly news that modern liberals view themselves as the party of science and experts. This is reflected in popular liberal slogans (“I believe in science”, “Follow the science”, “Trust the experts”, etc.), in the willingness of elite scientific and academic institutions to align themselves with liberal politics, and in many liberals’ eager embrace of highly counterintuitive ideas originating within universities—for example, concerning omnipresent but invisible (i.e., implicit and systemic) forms of oppression, or the fluidity and self-construction of gender. Perhaps most tellingly, it is reflected in the blue tribe’s popular “post-truth” analysis that rejecting the authority of credentialed experts amounts to rejecting truth itself.

Post-truth

Before turning to Friedman’s account of the history leading to America’s epistemological crisis, it is helpful to consider this “post-truth” analysis favoured by many experts within the blue tribe first.

Although this analysis comes in different forms, the core idea is that before Trump and his precursors (e.g., the Tea Party) took over the Republican Party and ushered in the “post-truth era”, America inhabited a golden age of objectivity, the truth era.

Of course, there was some disagreement within the truth era, and occasional epistemic fuck-ups, such as invading countries based on false information and blowing up the world economy based on false economic theories. Nevertheless, according to the post-truth analysis, this occurred against a background of substantial consensus, deference to experts, and respect for truth. Influential people did not brazenly lie about crowd sizes, suggest Satanic paedophiles run the government, or fabricate an epidemic of pet-eating immigrants.

Post-truth scholars typically trace the beginning of the decline of this golden age to the propagandistic activities of tobacco companies and fossil fuel companies. These companies purposefully sought to create doubt about expert consensus on the harms of smoking and the reality of climate change, which they achieved primarily by influencing the right-wing media ecosystem that emerged in the late twentieth century (especially Fox News). This then paved the way for a more general right-wing attack on science, experts, and truth itself.

As Lee McIntyre puts it in On Disinformation, a representative treatment of this topic favourably reviewed by almost every prestigious liberal outlet,

“One imagines some ambitious, orange-haired politician making the cynical leap of inference from cigarettes and global warming to other fact-based beliefs: “Why, if they can get away with lying about that, I can lie about anything at all.” And he did.”

Subscribe now

Friedman’s alternative history

Friedman tells a different story.

Although there was a kind of “golden age” in American politics, it was not a golden age of truth or objectivity but rather a golden age of political consensus: the post-New Deal consensus. According to Friedman, this was overwhelmingly a consensus of establishment liberals, indicated by the political dominance of the Democratic Party throughout this time, the popularity of liberals even within the GOP, and the hegemony of establishment liberalism within leading cultural institutions, including academia, media, and the arts.

Of course, this liberal dominance was not absolute. For example, the Republicans occasionally elected a president due to the personal charisma of their nominees or events that undermined the popularity of Democratic candidates. Moreover, there was some dissent from establishment liberal ideas within elite political culture, such as William F. Buckley’s National Review and, later, towards the end of the golden age, the Chicago School’s mainstreaming of right-wing libertarianism, primarily through the efforts of Milton Friedman.

Nevertheless, the several decades of the middle-twentieth century were overwhelmingly a period of establishment liberal hegemony. It is this period of “epistemological complacency”, writes Friedman, “that the post-truth discourse mourns, for post-truth scholars mistake agreement—agreement among experts, and agreement with experts by nonexperts—as a sign of truth.”

Of course, this “golden age” eventually broke down. To understand this process, Friedman highlights the trajectory of two segments of American society that were marginalized and excluded by the liberal establishment consensus.

The emergence of the right

Friedman dates the “beginning of the end of the Golden Age” to 1987 when the FCC repealed the Fairness Doctrine, which had required those with broadcast licenses to present political controversies in a fair and balanced way.

This paved the way for the emergence of a thriving right-wing media ecosystem that catered to a large segment of American society excluded from the establishment liberal consensus. This included highly influential talk radio hosts such as Rush Limbaugh, who—as an article in the Wall Street Journal puts it—”was the first man to proclaim himself liberated from the East Germany of liberal media domination.” Of course, it later included Fox News, which launched in 1996.

One aspect of this right-wing media ecosystem involved the positive affirmation, celebration, and justification of commonsense attachments to faith, flag, and free markets. However, an equally important aspect was an obsession with liberal bias in mainstream media and other cultural institutions, such as academia and Hollywood. (Fox’s original slogan was “Fair and Balanced”, an explicit rebuke to mainstream liberal media, which was depicted as unfair and unbalanced).

According to Friedman, the attachment to naive first-person realism led conservatives to interpret mainstream liberal bias as sinister. Because "commonsense” beliefs are self-evidently true, the establishment was not just mistaken but deliberately biased:

“The media and other elites who got reality wrong, according to the populist conservatives, did so knowingly, as these elites, like everyone else, had access to the self-evident truths that they claimed to reject. The charge of media bias, then, led to the conclusion that liberal elites were engaged in a conspiracy against the truth in the service of self-serving political ends.”

Once the growing influence of the conservative media ecosystem became apparent to establishment liberals, they reacted similarly.

Especially given the prominence of climate change scepticism within this new media ecosystem, such liberals, as third-person naive realists, assumed conservatives must be aware of the self-evident truth of climate science. Given this, they treated the rejection of climate science as a kind of mass deception and denial.

Subscribe now

The integration of the radical left

An equally important part of this history focuses on another segment of American society marginalised by the post-New Deal consensus: the radical left. As a fundamentally establishment consensus, the “golden age” was just as hostile to anti-establishment, left-wing radicals as to right-wing populists. Such radicals viewed the liberal establishment not as a bastion of truth and objectivity but as a propagandistic tool of capitalism, American imperialism, and—especially according to radicals near the end of the “golden age”—white supremacist, patriarchal, and heteronormative hegemony.

Nevertheless, left-wing radicals went on a very different journey to right-wing populists. Instead of setting up their own institutions and alternative epistemic universe, they gradually integrated into establishment institutions. (Friedman traces the beginnings of this “long march” of left-wing radicals through institutions to a vast number of job openings in American universities in the late 1950s, which enabled left-wing radicals to enter the academy in large numbers).

Of course, the relationship between establishment liberals and left-wing radicals has not always been a happy marriage. However, Friedman suggests that the cumulative effect of this process has been a transformation of establishment liberal institutions. Popular forms of progressive radicalism have moderated as they sought to transform these institutions, and these institutions have become more explicitly aligned with left-wing, progressive politics. (For an illustration of this dynamic, check out Scientific American’s explicit endorsement of Kamala Harris for president).

This process has dramatically exacerbated the mutual alienation and hostility felt between America’s two tribes:

“The resulting identification of academic expertise with an activist left-wing orthodoxy, which is now officially proclaimed on university websites, in college admissions materials, and in first-year orientation programming, only serves to confirm, on the right, the suspicion that “expertise” is an ideological sham.”

Of course, the blue tribe—the party of science and experts—does not view things this way. As naive realists themselves, they view the connection between expertise and progressive politics as an objective response to the self-evident fact that conservatives have abandoned truth.

Trump, Floyd, and the Acceleration of Epistemic Polarization

Friedman’s story ends with two events: the 2016 election of Donald Trump and the killing of George Floyd.

“Trump and his supporters were accused of racism as soon as he announced his candidacy. Therefore, his election, coupled with the specter of widespread police violence against black men, solidified the conviction, on the left and in the mainstream, that little or no racial progress had been made since the Civil Rights movement. After Floyd’s death, this conviction led to a veritable anti-racist revolution that swept every major cultural institution, from universities to art museums and children’s book publishing.”

According to Friedman, Trump voters have not just felt insulted by these pervasive accusations of racism and the new ways of conceptualising and understanding racism within establishment institutions. They have also felt baffled. Among those for whom “common sense” includes the beliefs that they are colorblind, that racial progress has been substantial, and that “reverse racism” is just as bad as anti-black racism, the explicit repudiation of these ideas among the establishment looks insane.

Evaluating the analysis

Is Friedman’s analysis correct?

Clearly, any general story like this can, at best, amount to a simplified, coarse-grained depiction of far more complex events and processes. Given this, rather than focusing too much on the specifics of Friedman’s historical account, I will identify some weaknesses in his general approach.

This approach is rooted in a fundamental methodological assumption: we should treat people’s beliefs on their own terms when attempting to analyse politics. That is, rather than trying to debunk or deconstruct beliefs by tracing them to underlying social, political, economic, material, or psychological causes, we should treat people’s worldviews seriously and sympathetically as an independent explanatory domain. This is why Friedman speculates that America’s “present political crisis is nothing but an epistemological crisis.”

In this sense, Friedman’s analysis involves a fairly extreme “idealist” approach to the role of ideas in society. Roughly, idealists treat ideas as an autonomous realm that explains human behaviours and social organisation, in contrast to so-called “materialist” approaches that explain ideas in terms of underlying, more fundamental factors. (Marx’s claim that ideas are merely parts of a society’s “superstructure” determined by its economic “base” makes him the archetypal materialist in this sense.) Of course, these are two ends of a continuum rather than a simple dichotomy.

When combined with his radical skepticism and excessive intellectual charity, Friedman’s idealism produces various weaknesses in his account.

Against idealism

First, although Friedman insightfully describes some real trends in American belief systems, ideas do not come from nowhere. For example, any analysis of the evolution of popular ideologies embraced by America’s political tribes must consider the shifting alliance structure of American politics: the distinct groups in society that support the two main parties.

As is well-documented, one reason for the strangely non-polarized aspect of mid-twentieth-century American politics is that both parties were effectively united in the goal of maintaining racial segregation and discrimination, especially in the South. It was only once the Democrats passed major civil rights legislation in the 1960s that American politics began its great ideological and social re-sorting, with black voters flocking to the Democrats and many white voters embracing the Republicans. Describing this process, David Pinsof and colleagues observe three other major political realignments:

[1] The Republican Party took ownership of the pro-life, evangelical movement, causing Christian traditionalists to move into the Republican Party and secular feminists to move into the Democratic Party. [2] Influxes of immigrants from Latin America—coupled with urbanization and the decline of manufacturing work—gave rise to a rural, white underclass who attributed their declining status to immigration and globalization. [3] At the same time, expanding college enrollment produced a new upper class of highly educated “knowledge workers”, while large corporations commanded an increasingly greater share of wealth and political power. These trends resulted in competition and resentment between intellectual elites (e.g., highly educated professionals) and business elites (e.g., wealthy corporate executives). In other words, the lower class split apart based on ethnic rivalries, while the upper class split apart based on status rivalries, thereby weakening the historical link between partisanship and class.”

You cannot understand the prevalent belief systems in American society and the two tribes’ attitudes towards expertise without engaging with these forces. For example, the fact that the Democratic Party involves a coalition between highly educated white professionals and racial minorities illuminates the degree to which modern liberalism in the US combines deference to credentialed experts with a strong focus on anti-racist politics. Likewise, the strange coalition between rural, uneducated, white, socially conservative Americans and business elites within the Republican Party illuminates why American conservatism became so strongly pro-market in the late twentieth century.

To be clear, ideologies cannot be reduced to alliance structures, not least because the groups people identify with and support depend on which ideas they embrace. Nevertheless, they do not exist in a wholly autonomous domain either.

Subscribe now

Too much charity

In addition, although Friedman’s commitment to intellectual charity—to treating people as sincere, rational agents rather than dismissing their views as consequences of deception, self-deception, or irrationality—is refreshing, it is also excessive.

He is right that naive realism is false. The truth is not self-evident, and even perfectly rational, well-meaning people can acquire very different worldviews based on the information they obtain from others and how they interpret it. Nevertheless, it is a fallacy to infer from the fact that error and disagreement can be caused by purely honest, rational processes that they always are.

For one thing, lying and propaganda—in the preferred language of the blue tribe today, “disinformation campaigns”—have played an undeniable role in American politics. Corporations peddle self-serving information and attempt to influence public opinion, as do political and cultural elites.

Moreover, humans are not disinterested truth seekers. We engage in motivated reasoning. We advocate for beliefs and narratives that promote our and our favourite groups’ interests and adopt ideas that win us trust and status within our ingroup. Being humans, these motives powerfully distort how Americans view the world.

They also shape the social dynamics of America’s conflicting political tribes, which transform partisan narratives into sacred beliefs, reward pundits and intellectuals who rationalise and defend those beliefs, and ostracize and cancel heretics who challenge them.

Friedman is correct in saying that many people exaggerate the role of such factors. Moreover, much discourse about propaganda, motivated reasoning, and groupthink is itself a form of propagandistic motivated reasoning, treating these issues as if they were restricted to only one of America’s tribes.

Nevertheless, you cannot understand American politics and its epistemological character if you treat people as dispassionate, truth-seeking robots. Humans are strategic, status-seeking primates who view reality in ways biased by self-interest and tribal allegiances.

Subscribe now

America’s belief systems

Once you understand the role of political alliances and motivated reasoning in shaping political belief systems, it also becomes clear that Friedman’s claim that conservatives and liberals embrace different forms of naive realism is not entirely correct.

Although it is true that the right and left embrace naive realism and adopt very different attitudes towards credentialed experts and establishment epistemic institutions (science, universities, public health, mainstream media, etc.), their attitudes towards experts are not always consistent.

For example, although the left is happy to trust experts on topics like climate change, vaccines, and public health guidance, these issues all align with its political agenda. On other topics (sex differences, IQ, behavioural genetics, mainstream economics, and so on), there is often a willingness to dismiss or ignore expert consensus. For example, a recent article in The Atlantic defends Kamala Harris’s proposed price-gouging ban with the title, “Sometimes You Just Have to Ignore the Economists.” And even on topics like climate change, liberals tend to be highly selective in which aspects of expert consensus they defer to.

Admittedly, this is made easier because one can always find academics willing to endorse preferred liberal policies or narratives, not least because of the dominance of liberals within these institutions. Nevertheless, the point is that the story is more complex—and more biased—than a default attitude of deference towards credentialed experts.

Moreover, in some cases, liberals are willing to abandon deference towards “expert” knowledge altogether if doing so aligns with political goals, as with the celebration of “lived experience” over statistical data when the lived experiences align with progressive narratives.

Something similar applies to the right, where conservatives are often happy to instrumentalize experts when their conclusions align with conservative causes and narratives. However, Friedman’s analysis is closer to the mark here because the neglect and contempt of intellectual thought increasingly look all-encompassing since Trump took over the conservative movement.

Passing epistemic judgment

Finally, Friedman’s analysis of America’s epistemological crisis attempts to avoid any assessment of the relative epistemic quality of its two tribes—that is, the degree to which their belief systems accurately map reality.

To some degree, this is understandable. Given Friedman’s rejection of naive realism, he denies that ideologies can be evaluated as categorically true or false. Instead, following Walter Lippmann, he thinks of such belief systems as selective, low-resolution models of reality. This means he treats both political tribes in the US as embracing “a different set of interpretive frameworks that determines how and what it sees of reality.”

Nevertheless, one can reject naive realism and acknowledge the possibility of multiple perspectives without thinking all perspectives are equally accurate. At times, Friedman seems to elide this distinction.

Of course, one can never step outside one’s belief system and evaluate its correspondence to reality. Any evaluation of a set of beliefs will inevitably draw on one’s own beliefs, which are highly vulnerable to error and partiality for the reasons Friedman identifies. However, this is a reason to be careful and embrace fallibilism, not radical skepticism.

With this in mind, I will make two general observations about America’s epistemological crisis.

Sectarian misperceptions

First, due to the highly polarized and sectarian nature of modern American politics, both sides view reality in ways that are distorted by partisanship and group allegiances. This biases judgment at the individual level. However, it has also created thriving rationalization markets in which members of the two tribes compete to win status and financial rewards by justifying and defending their faction’s favoured narratives.

Although this is common to all politics, it is especially toxic in America because the passionate partisans and pundits of both tribes increasingly inhabit distinct media ecosystems. With declining intergroup communication, the result seems to be growing dogmatism and radicalisation.

The failure modes of first-person and third-person naive realism

Second, Friedman’s analysis of the two tribes’ attitudes towards experts illuminates their different failure modes.

Even though the blue tribe often approaches experts with selectivity and flexibility, its general deference to establishment epistemic institutions produces distinctive errors and blind spots. The simple reason is that these institutions are far from perfect. Outside the hard sciences, many ideas advanced and legitimised within science and academia are simplistic, selective, biased, and unreliable.

The replication crisis is one indication of this. There were also many well-documented problems with public health authorities during the Covid-19 pandemic. However, there are countless others. Experts are often overconfident and wrong. Whole bodies of putatively scientific knowledge are commonly built on sand. These problems are exacerbated by a situation where the line between progressive activism and science is not always clear (and sometimes wilfully blurred). And in many ways, things are even worse within establishment liberal media.

These and numerous other factors ensure that the blue tribe’s picture of reality is frequently biased, selective, or plain wrong. Moreover, without these evident problems with America’s epistemic institutions, the red tribe’s proud rejection of such institutions would probably not be possible.

Nevertheless, the blue tribe’s problems are much less severe than those confronting the red tribe.

The Republican Party and conservative media today have become almost fully unmoored from reality. Utterly baseless lies, fabrications, conspiracy theories, and absurdities run rampant. Nearly everything that comes out of Trump’s mouth is a lie or exaggeration. And remarkably, this situation seems to worsen as time passes.

Few things could illustrate this more powerfully than Trump’s preposterous, evidence-free, racist claim that immigrants are eating Americans’ pets en masse, something which most Republicans apparently believed and Tucker Carlson celebrated as “awesome” because it “makes all the right people mad.”

This is the grain of truth in the blue tribe’s “post-truth” analysis of the modern Republican Party. However, the problem is not that the red tribe has wholly abandoned any concern with truth. The problem is that without knowledge-generating institutions and their norms and procedures (e.g., in science and professional journalism), caring about the truth achieves nothing. The consequence is instead a reversion to an epistemic state of nature in which ignorance, error, and tribal narratives are the default state.

In other words, for all the problems with America’s knowledge-generating institutions, these institutions evolved over centuries for a reason. If you reject them wholesale, the result is not liberation from bias and delusion; it is the complete capitulation to them.

On the one hand, the educational potential of AI is staggering. Modern large language models like ChatGPT offer incredible opportunities for 24/7 personal tutoring on any topic you might want to learn about, as well as many other benefits that would have seemed like science fiction only a few years ago. One of the really fun parts of this conversation was discussing how we personally use AI to enhance our learning, reading, and thinking.

On the other hand, AI has clearly blown up the logic of teaching and assessment across our educational institutions, which were not designed for a world in which students have access to machines that are much better at writing and many forms of problem-solving than they are.

And yet… there has been very little adaptation.

The most obvious example is that many universities still use take-home essays to assess students.

This is insane.

We discuss this and many other topics in this conversation, including:

• How should schools and colleges adapt to a world with LLMs?

• How AI might exacerbate certain inequalities.

• Whether AI-driven automation of knowledge work undermines the value of the skills that schools and colleges teach today.

• How LLMs might make people dumber.

Links

• John Burn-Murdoch, Financial Times, Have Humans Passed Peak Brain Power?

• James Walsh, New York Magazine, Everyone Is Cheating Their Way Through College

• Rose Horowitch, The Atlantic, Accommodation Nation

Transcript

Note: this transcript is AI-generated and may contain mistakes.

Dan Williams

Welcome everyone. I’m Dan Williams, and I’m back with my friend and co-conspirator, Henry Shevlin. Today we’re going to be talking about a topic which is close to both of our hearts as academics who have spent far too long in educational institutions: the impact of AI on education and learning in general, but also more specifically on the institutions—the schools and universities—that function to provide education.

There’s a fairly simple starting point for this episode, which is that the way we currently do education was obviously not built for a world in which students have access to these absolutely amazing writing and problem-solving machines twenty-four seven. And yet, for the most part, it seems like many educational institutions are just carrying on with business as usual.

On the one hand, the opportunities associated with AI are absolutely enormous. Every student has access twenty-four seven to a personal tutor that can provide tailored information, tailored feedback, tailored quizzes, flashcards, visualizations, diagrams, and so on. On the other hand, we’ve quietly blown up the logic of assessment and lots of the ways in which we traditionally educate students—most obviously with the fact that many institutions, universities specifically, still use take-home essays as a mode of assessment, which, at least in my view (and I’m interested to hear what Henry thinks), is absolutely insane.

So what we’re going to be talking about in this episode are a few general questions. Firstly, what’s the overall educational potential when it comes to AI, including outside of formal institutions? What are the actual effects that AI is having on students and on these institutions? How should schools and universities respond? And then most generally, should we think of AI as a kind of crisis—a sort of extinction-level threat for our current educational institutions—or as an opportunity, or as both?

So Henry, maybe we can start with an opening question: in your view, what is the educational potential of AI?

Henry Shevlin

I think the educational potential is insane. I almost think that if you were an alien species looking at Earth, looking at these things called LLMs, and asking why we developed these things in the first place—without having the history of it—you’d think, “There’s got to be some kind of educational tool.” If you’ve read Neal Stephenson’s The Diamond Age, you see a prophecy of something a little bit like an LLM as an educational tool there.

I think AI in general, but LLMs specifically, are just amazingly well suited to serve as tutors and to buttress learning. Probably one key concept to establish right out the gate, because I find it very useful: some listeners may be familiar with something called Bloom’s two sigma problem. This is the name of an educational finding from the 1980s associated with Benjamin Bloom, one of the most prominent educational psychologists of the 20th century, known for things like Bloom’s taxonomy of learning.

Basically, he did a mini meta-analysis looking at the impact of one-to-one tutoring compared to group tuition. He found that the impact of one-to-one tutoring on mastery and retention of material was two standard deviations, which is colossal—bigger than basically any other educational intervention we know of. Just for context, one of the most challenging and widely discussed educational achievement gaps in the US, the gap between black and white students, is roughly one standard deviation. So this is twice that size.

Now, worth flagging, there’s been a lot of controversy and deeper analysis of that initial paper by Bloom. For example, the students in the tutoring groups were mostly looking at students who had a two-week crammer course versus students who had been learning all year, so there were probably recency effects. He was only looking at two fairly small-scale studies. Other studies looking at the impact of private tutoring versus group tuition have found big effects, even if not quite two standard deviations. And this makes absolute intuitive sense—there’s a reason the rich and famous and powerful like to get private tutors for their kids.

Dan Williams

Yeah.

Henry Shevlin

There’s a reason why Philip II of Macedon got a tutor for Alexander. And more broadly, I think we can both attest as products of the Oxbridge system: one of the key features of Oxford and Cambridge is that they have one-to-one tutorials (or “supervisions,” as the tabs call them). This is a really powerful learning method.

So even if it’s not two standard deviations from private tuition, it’s a big effect. Now, people might be saying, “Hang on, that’s private tuition by humans. How do we know if LLMs can replicate the same kind of benefits?” It’s a very fair question. In principle, the idea is that if it’s just a matter of having someone deal with students’ individual learning needs, work through their specific problems, figure out exactly what they’re misunderstanding and where they need help, there’s no reason a sufficiently fine-tuned LLM couldn’t do that.

I think this is the reason Bloom called it the “two sigma problem”—it was assumed that obviously you can’t give every child in America or the UK a private tutor. But if LLMs can capture those goods, everyone could have access to a private LLM tutor.

That said, I think the counter-argument is that even if we take something like a two standard deviation effect size on learning and mastery at face value, there are things a human tutor brings to the table that an AI tutor couldn’t. Social motivation, for one. I don’t know about your view, but my view is that a huge proportion of education is about creating the right motivational scaffolds for learning. Sitting there talking to a chat window is a very different social experience from sitting with a brilliant young person who’s there to inspire you. Likewise, I think it’s far easier to alt-tab out of a ChatGPT tutor window and play some League of Legends instead, whereas if you’re sitting in a room with a slightly scary Oxford professor asking you questions, you can’t duck out of that so easily.

So I think there are various reasons why we probably shouldn’t expect LLM tutors to be as good as human private tutors. But I think the potential there is still massive. We don’t know exactly how big the potential is, but I think there’s good reason to be very excited about it. And personally, I find LLMs have been an absolute game-changer in my ability to rapidly learn about new subjects, get up to speed, correct errors. In a lot of domains, we all have questions we’re a little bit scared to ask because we think, “Is this just a basic misunderstanding?”

Dan Williams

Yeah.

Henry Shevlin

Anecdotally, I know so many people—and have experienced firsthand—so many game-changing benefits in learning from LLMs. But at the same time, there’s still a lot of uncertainty about exactly how much they can replicate the benefits of private tutors. Very exciting either way.

Dan Williams

I think there’s an issue here, which is: what is the potential of this technology for learning? And then there’s a separate question about what the real-world impact of the technology on learning is actually going to be. That might be mediated by the social structures people find themselves in, and also their level of conscientiousness and their own motivations. We should return to this later on. Often with technology, you find that it’s really going to benefit people who are strongly self-motivated and really conscientious. Even with the social media age—we live in a kind of informational golden age if you’re sufficiently self-motivated and have sufficient willpower and conscientiousness to seek out and engage with the highest quality content. In reality, lots of people spend their time watching TikTok shorts, where the informational quality is not so great.

But let’s stick with the potential of AI before we move on to the real-world impact and how this is going to interact with people’s actual motivations and the social structures they find themselves in.

So the most obvious respect in which this technology is transformative, as you said, is this personal tutoring component. Maybe we could be a bit more granular. We’re both people who benefit enormously from this technology. I think I would have also benefited enormously from this technology if I’d had it when I was a teenager. I remember, for example, when I was a teenager, I did what many people do when they feel like they want to become informed about the world: I got a subscription to The Economist when I was 14 or 15. And I would work my way through it, dutifully trying to read all of the main articles.

Henry Shevlin

God.

Dan Williams

And at the time I’d think, “What’s fiscal and monetary policy? I don’t fully understand the political system in Germany.” But if I’d had ChatGPT at the time, I could have just asked, “Explain to me the difference between fiscal and monetary policy.”

Henry Shevlin

I’m cringing because I did exactly the same thing. The Economist and New Scientist were the two cornerstones of my teenage education.

Dan Williams

So let’s maybe talk about how we use it, and at least in principle, if people had the motivation and the structure to encourage that motivation, how the technology could be beneficial for the process of learning. For example, how do you personally use this technology to enhance your ability to acquire and integrate and process information?

Henry Shevlin

I think one useful way to introduce this is to think about other kinds of sources of information and what ChatGPT adds. As a kid and as a teenager, I remember very vividly—I think I was about 10 years old when we got our first Microsoft Encarta CD-ROM encyclopedia. It blew my mind; I could do research on a bunch of topics. Some of them even had very grainy pixelated videos. It was great fun. And obviously the internet adds a further layer to your ability to do research. I’m also the kind of person who, even before the launch of ChatGPT, at any given time had about 30 different tabs of Wikipedia open.

So if you’re the kind of person who is interested and curious about the world, we live in an informational golden age. Our ability to learn about things has been improving; our tools for learning about the world have been improving. So what does ChatGPT and LLMs add on top of that?

First, I often find that even Wikipedia entries can be very hard to get my head around, particularly if I’m trying to do stuff in areas I’m not so good at. If I’m looking at some concepts in physics or maths—maths is particularly hilarious here. If you look up a definition of a mathematical concept, it’s completely interdefined in terms of other mathematical concepts. Absolute nightmare. Even philosophy can be the same: “What is constructivism? Constructivism is a type of meta-ethical theory that blah, blah, blah.” It can quickly get lost in a sea of jargon where all the terms are interdefined. Whereas you can just ask ChatGPT, “I’m really struggling. What is Minkowski spacetime? Please explain. ELI5—explain to me like I’m five.”

So in terms of getting basic introductions to complex concepts, being able to ask questions as you go—this is huge. Being able to check your knowledge and say, “Is this concept like this? Am I right? Am I misunderstanding this?” Being able to draw together disparate threads from topics—this is something that’s basically impossible to do prior to LLMs unless you get lucky and find the right article. So if I ask, “To what extent did Livy’s portrayal of Tullus Hostilius in his book on the foundation of Rome draw inspiration from the figure of Sulla?” (This is a specific example because I wrote an essay about it.) These kinds of questions where you’re drawing together different threads and asking, “Is there a connection between these two things, or am I just free-associating? Is this thing a bit like this other thing?”

Dan Williams

Yeah.

Henry Shevlin

These kinds of questions—you can just ask them. Other really good things you can do, getting into more structured educational usage: you can ask for needs analysis. Recently I was trying to get up to speed on chemistry—chemistry was my weakest science at high school. I said, “ChatGPT, I want you to ask me 30 questions about different areas of chemistry. Assume a solid high school level of understanding and identify gaps in my knowledge. On that basis, I want you to come up with a 10-lesson plan to try and plug those gaps.” And then you can just talk through it. I did a little mini chemistry course over about 30 or 40 prompts. So that’s a slightly more profound or interesting use.

Another really powerful domain is language learning. I’m an obsessive language learner; at any one time, I usually have a couple on the go.

Dan Williams

Hmm.

Henry Shevlin

Duolingo—I had a 1,200-day streak at one point—but it sucks, I’ll be honest, for actually improving fluency. It’s very good for habit formation, but it doesn’t really teach grammar concepts very well. It doesn’t build conversational proficiency very well. It’s okay for learning vocab. But LLMs used in the right way can be fantastically powerful tools for this.

Particularly with grammar concepts, you’ve often got to grok them—intuitively understand them. So being able to say, “Am I right in thinking it works like this? How about this kind of sentence? Does the same rule apply?” Or when learning a language, you’ll often encounter a weird sentence whose grammar you don’t understand. This is something you couldn’t really do prior to ChatGPT in an automated fashion: “Can you explain the grammar of this sentence to me? I just don’t get it.”

Also, Gemini and ChatGPT both have really good voice modes that are polyglot. So you can say, “ChatGPT, for the next five minutes, I want to speak in Japanese” or “I want to speak in Gaeilge. Bear in mind my language level is low, my vocabulary is limited. Try not to use any constructions besides these.” Or even, “Let’s have a conversation practising indirect question formation in German.” You can do these really tailored, specific lessons.

I’ll flag that language learning is one area where in particular the applications and utility of LLMs are just so powerful and so straightforward. But it’s funny—I’ve yet to see the perfect LLM-powered language learning app. Someone might comment on this video, “Have you checked out X?” But I’m sure in the next couple of years, someone is going to make a billion-dollar company on that basis.

Dan Williams

Surely, yes. Just to add another couple of things in terms of how I use it, which actually sounds very close to how you use this technology. One thing is: I think an absolutely essential part of thinking is writing. People often assume that with writing, you’re just expressing your thoughts. Whereas actually, no—in the process of writing, you are thinking.

One of the things that’s really great as a writer—I’m an academic, so I write academic research; I’m also a blogger and I write for general audiences—is to write things and say, “Give me the three strongest objections to what I’ve written.” And often the objections are actually really good. That’s an incredible opportunity because historically, if you wanted to get an external mind to critique and scrutinise what you’ve written, you’d have to find another human being, and they’re going to have limited attention. That’s really challenging. Whereas now you can get that instantly.

I also find that now when I’m reading a book—and I think reading books is absolutely essential if you do it the right way for engaging with the world and learning about the world—I’ll do the thing you’ve already mentioned: if there’s anything I don’t understand or don’t feel like I’ve got a good grip on, I’ll ask ChatGPT to provide a summary or explain it in simpler terms. But I’ll also often upload a PDF of the book when I can get it and think, “Here’s my current understanding of chapter seven. Can you evaluate the extent to which I’ve really understood it and provide feedback on something I’m missing?”

What you can also do—and I find Gemini is much better at this than ChatGPT—is ask it to generate a set of flashcards on the material, then take the flashcards it’s generated and ask it to create a file for Anki (which is a flashcard program) that you can import directly and use to test yourself on the knowledge over time. In principle, you could have done that prior to the age of AI, but the ease and pace with which you can do it today is absolutely transformative in terms of your ability to really quickly master material. So those are just a few things off the top of my head. I’m sure there are many other uses.

Henry Shevlin

That Anki suggestion is gold. I use Anki, and just to be clear to anyone not familiar: it is one of the best educational tips I can ever recommend. In any situation where you need to remember a mapping from some X to some Y—that could be learning vocabulary, mapping an English word to a Japanese word; it could be mapping a historical event to a date; mapping an idea to a thinker; or, one use case for me, mapping a face to a name (it doesn’t just need to be words).

One trick I used to do with my students: we’d have 40 students join each term in our educational programs. I’d create a quick flashcard deck with their photos (which they submit to the university system) and their names, and you can memorise their names in half an hour to an hour. It really does feel like, if you’ve never used it before, a cheat code for memory. It’s astonishing.

Dan Williams

Yeah.

Henry Shevlin

But I have not used Gemini for creating Anki decks. This is genius. And I think this illustrates a broader point: we’re still figuring out, and people are stumbling upon, really powerful educational or learning use cases for these things all the time. Even in this conversation—I think we’re both pretty power users of these systems—but I just picked up something from you right there. I have these conversations all the time: “Great, that’s a brilliant use case I hadn’t thought of.”

One thing I’ll also flag that maybe more people could play around with is really leaning into voice mode a bit more. Voice mode is currently not in the best shape—well, it’s the best shape it’s ever been, but I think we’re still ironing out some wrinkles on ChatGPT. Still, if I’m on a long car journey (I drive a lot for work, often to the airport), I’ll basically give a mini version of my talk to ChatGPT as we’re driving along. I’ll say, “Here’s the talk I’m going to be giving. What are some objections I might run into?” And we’ll have a nice discussion about the talk.

Or sometimes I’m just driving back, a bit bored, I’ve listened to all my favourite podcasts. I’ll say, “ChatGPT, give me a brief primer on the key figures in Byzantine history,” or “Give me an introduction to the early history of quantum mechanics.” Then I’ll ask follow-up questions. It’s like an interactive podcast.

Dan Williams

That’s awesome. One thing that’s really coming across in this conversation is the extent to which we’re massive nerds and potentially massively unrepresentative of ordinary people.

Okay, maybe we can move on. We both agree that the potential of this for learning material, mastering material, improving your ability to think, understand, and know the world is immense.

But obviously there’s a big gap between the potential of a technology in principle and how it actually manifests in the world. I mentioned the internet generally, social media specifically, as an obvious illustration of that. Even though I think a lot of the discourse surrounding social media is quite alarmist, at the same time it does seem to have quite negative consequences with certain things and among certain populations—specifically, I think, those people who don’t have particularly good impulse control. Social media is really a kind of hostile technology for such people.

Henry Shevlin

I’d also add online dating as another example of a technology that sounds like it should be so good on paper. And at various points it has been really good—I met my wife on OkCupid back in 2012. But it seems like what’s happened over the last 10 years, speaking to friends who are still using the various dating apps, is it’s almost a tragedy of the commons situation. Something has gone very wrong in terms of the incentives so that it’s now just an unpleasant experience. Straight men who use it say they have to send hundreds of messages to get a response. Women who use it say they just get constantly spammed with low-effort messages. I give that as another example: we’ve built these amazing matching algorithms—why isn’t dating a solved problem right now? It turns out there can be negative, unexpected consequences with these technologies.

Dan Williams

That’s a great example. So what’s your sense of what the actual real-world impact of AI is on students and teachers and educational institutions at the moment?

Henry Shevlin

This is probably a good time for our disclaimers. I’m actually education director of CFI with oversight of our two grad programs. So everything I say in what follows is me speaking strictly in my own person rather than in my professional role.

Dan Williams

Can I just quickly clarify—CFI is the Centre for the Future of Intelligence, where you work at the University of Cambridge. Just for those who didn’t know the acronym.

Henry Shevlin

Exactly, that’s helpful. The Centre for the Future of Intelligence, University of Cambridge. I’m the education director, with ultimate oversight of our 150-odd grad students. But we only have grad students, and I think this means my perspective on the impact of AI on education is quite different from where I think the real catastrophic or chaotic impacts are happening. Grad students are a very special case; they tend to have high degrees of intrinsic motivation. The incentive structures for grad students—where they’re directly paying for the course themselves in many cases, or for our part-time course they’re being paid for by employers who expect results—all of this creates a quite different environment.

So when I talk about impacts on education, I’m going to be mainly focusing on undergrad education and high school. These are areas where I’m not speaking from first-hand experience, but from many conversations with colleagues who teach undergrads. I don’t really teach undergrads, but lots of colleagues do. And several of my very closest friends are teachers in high schools and, in a couple of cases, primary schools. So I’m drawing on what they’re seeing.

Dan Williams

Can I also give my own disclaimer: as with every other topic we focus on, I’m an academic—an assistant professor at the University of Sussex—and I’m giving my personal opinions, not the opinions of the institution I work for. Okay, sorry to cut you off.

Henry Shevlin

No, excellent. Our respective arses thoroughly covered.

So with that in mind, there’s a phenomenal piece by James Walsh in NY Mag from back in May this year called “Everyone is Cheating Their Way Through College.” It’s a beautiful piece of long-form journalism. Can’t recommend it enough—absolutely exhilarating and horrifying—talking about the impact of ChatGPT and other LLMs on education.

“Complex and mostly bad” is the short answer for what the actual short-term impacts of LLMs have been. When ChatGPT launched, I said something similar to what you said at the opening of the show: the take-home essay assignment is dead for high school, and pretty soon it’ll be dead for university. And then, yeah, pretty soon it was dead for university.

So I think that’s the most straightforward initial impact: we can no longer assign graded take-home essay assignments with any real confidence, particularly for high school and undergrad students, because it’s just so easy to get ChatGPT to do it. I believe that people, even with the best intentions, are responsive to incentives. And if you can produce an essay that’s—honestly, these days with contemporary language models—very good quality, particularly at high school level, even undergrad level, if ChatGPT can just do something as good or better than you, then why bother putting in the work? If you’re hungover, or there’s a really cool party you want to attend, or you’re working a second job—we shouldn’t assume all students are living lives of leisure; lots of them are struggling to pay the rent.

So with all these incentives in place, no surprise that basically, as the article says, everyone is cheating their way through college. And I’m kind of appalled. I was in California a couple of weeks ago, just chatting to some students at a community college. One of them was a nursing student, and she said, “Yeah, I’m learning nothing at university. ChatGPT writes all my assignments. Ha ha ha ha ha.” And I was like, “Okay, got to be a bit careful about getting medical treatment—which hospital are you planning to work at?” But I think that’s symptomatic of broader problems.

Dan Williams

Of course. Maybe we can break that down point by point. We can say now with basically 100% certainty that large language models of the sort that exist today can write extremely good essays at undergraduate level. And I think there’s basically no way professors are going to be able to detect whether this has happened, at least if students are sufficiently skilful in how they do it.

I constantly come across academics who are still living in 2022, and they think, “Of course there are going to be these obvious hallucinations, and of course it’s going to be this mediocre essay.” I just think that’s not at all the reality of large language models today. If you know at all what you’re doing, you can delegate the task of writing to one of these large language models, which will produce an exceptional essay, and there’s no real way of knowing whether it’s been AI-generated.

There are tools which claim they can determine probabilistically whether an essay has been AI-generated. I don’t think those tools work, and I think they create all sorts of issues. If it’s not going to be 100% certain—which I think it basically never can be when it comes to AI-generated essays—then it becomes an absolute institutional nightmare trying to demonstrate that a student has used AI. I also think the incentives at universities, and indeed at schools more broadly, don’t encourage academics to really pursue this. It’s going to be an enormous amount of hassle, an enormous amount of extra work.

So I think what’s happening at the moment is that, to the extent universities and other institutions of higher education are using take-home essays specifically—but I’d say take-home assessments more broadly—to evaluate students, you’re basically evaluating how well they can cheat with AI. And I think that’s absolutely terrible.

Not just because, as you say, it means students aren’t actually encouraged to learn the material—they don’t really have an incentive to learn it. But one of the main functions of universities, of educational institutions more broadly, is credentialing. These are institutions that provide signals. They evaluate students according to their level of intelligence, their conscientiousness, and so on. The signal you get with a grade, and overall with your credential, is incredibly useful to prospective employers because they know: you got a first from Cambridge, you got a first from Bristol, whatever it might be. That’s a really good signal—not a perfect signal, but a pretty good signal—that you’re likely to be a good employee in certain domains.

To the extent that students are using AI to produce their assessment material, the signalling value of that just dissolves completely. That’s why, unless there’s urgent reform of the system and a move away from those sorts of take-home assessments, the problem here is not just that students aren’t learning things (which would be bad enough). I think it’s a kind of extinction-level threat for these institutions, because once it becomes clear that the grades you’re giving students don’t really provide any information about their intelligence or conscientiousness, then the social function of the institution dissolves completely. So that’s my take—do you disagree?

Henry Shevlin

No, completely agree. To pick up on a few thoughts: I imagine some people listening will say, “Yeah, but I can kind of tell when a student’s essay is written by ChatGPT.” I think a useful idea here is something I’ve heard called the toupee fallacy. People will say, “You can always tell when someone’s wearing a wig.” And you ask, “So what’s your reference set for that?” “Well, I often go out and I see something and it’s an obvious wig.” Okay, you’re seeing the obvious ones.

Dan Williams

Hmm.

Henry Shevlin

In other words, you don’t know. You can tell in cases where something is obviously a wig or obviously AI-generated. But you have no idea what the underlying ground truth is when it comes to the ones you can’t spot. You don’t have a way of identifying your rate of false negatives. I think that’s a really big problem.

Of course, anyone who marks papers will find occasional students who have left in “Sure, I can help you with that query” or “As a large language model trained by OpenAI...” But you know that’s the minority, and lots of essays you might assume are non-AI-generated are almost certainly AI-generated as well.

Relatedly, on the hallucinations point: this is obviously a big topic (we could probably do a whole episode on hallucinations), but rates of hallucinations have gone down dramatically. Particularly since search functionality was added to LLMs—they can go away and check things themselves. And also, you get the analogue of hallucinations just with student writing all the time. Even long before LLMs, students would falsely claim that Kant was a utilitarian or something, because they hadn’t properly understood the material or had misunderstood. So hallucinations are not a particularly good sign.

I think it’s basically impossible to tell. And as you emphasise, the false positive problem: even if you’re really confident an essay is AI-generated, good luck proving that. And is it really worth it for you as an educator to fight this tough battle with a student to bust them when everyone else is doing it? We just don’t have the incentives for educators or instructors to really enforce this.

Two other quick points. First, this creates huge problems not just with assessment but also with tracking students. This is something my high school teacher friends have really emphasised. It used to be, before ChatGPT, that essay assignments were a good way to keep track of which students were highly engaged with the class, which students were struggling, which students were really on top of the material. Whereas now we’ve seen a kind of normalisation effect where even the weakest students can turn in pretty solid essays courtesy of ChatGPT.

Dan Williams

Yeah.

Henry Shevlin

You’ve got no way of knowing which students need extra help versus which are already doing fine. That’s a big problem.

The final thing I’ll mention is that although take-home essay assignments are the ground zero of these negative effects, it covers other kinds of assignment as well. A colleague of mine who teaches at a big university (not Cambridge) was saying he’s been doing class presentations. Then he quickly realised students would generate the scripts for their presentations from ChatGPT. So he said, “Okay, we’re also going to partly grade them on Q&A, where students are graded on the questions they ask other presenters but also the responses they give.” And he said it quickly became clear: people would say, “Give me a moment to think about that question,” type into a computer, get the response from ChatGPT. Or people were using ChatGPT to generate questions.

I think there’s almost a generation of students for whom this is just their default way of approaching knowledge work, which I think is potentially a problem.

Dan Williams

The obvious solution, it seems, would be that you need modes of assessment where students can’t use AI—such as in-person pen-and-paper exams, such as oral vivas. And I do think that’s basically the direction these educational institutions are going to have to go.

However, that obviously creates issues. One is that there’s something incredibly valuable about learning how to write essays—not for everyone. Sometimes people like us, because of our interests and our passions and our profession, think it’s really important to have the ability to write long-form essays. And I totally understand that for many people, that’s a skillset which isn’t particularly useful for them. But in general, I do think for people who aspire to be engaged, thoughtful people, the skillset involved in writing long essays is incredibly valuable. So to the extent that the take-home essay and coursework disappear altogether, I think that’s a real issue in the sense that certain kinds of skills won’t be getting incentivised by our educational institutions.

But I also think it’s incredibly important that students learn how to use AI. That should be one of the main things educational institutions are providing to students these days: the ability to use AI effectively. And I think that skill is only going to become more important—in the economy, the labour market, and so on.

So on the one hand, it seems like large language models have made it basically impossible to have any kind of assessment other than in-person pen-and-paper tests or oral examinations. But on the other hand, to the extent we go down that route, many of the skills and knowledge you want students to acquire will no longer be encouraged and incentivised by educational institutions. That seems like a really big issue, and I have absolutely no idea what to do about it.

Henry Shevlin

Really good points. I’d agree you can do in-person essay exams. Most of my finals as an undergrad consisted of three-hour-long exams in which I had to write three essays. But that trains a very specific type of writing—quite an artificial one. It’s training your ability to write essays under tight pressure. If you want to do any kind of writing for a living, that’s only one of many skills you want.

If you’re producing writing you want people to read—whether it’s blogging or writing academic articles or scientific papers—you don’t typically write it under incredible time pressure where you’ve got to put out two and a half thousand words in three hours. You go through multiple drafts. You test those drafts with colleagues. There’s a whole bunch of writing skills that rely on the take-home component, the ability to think things through. And I don’t know how we test those.

Second, I completely agree that one of the things education is for is preparing people for knowledge work, and knowledge work these days is almost always going to involve the use of LLMs. So we should be training people to use them.

As to how we respond, my very flat-footed initial thought is we need to separate quite clearly: courses where LLM usage is trained and developed and built in as part of the assessment—where it’s assumed everyone will be using LLMs at multiple stages of the process and part of the skillset is using them effectively—versus other courses that say, “This is an LLM-free course; all assignments will be in-person vivas or in-person written exams.”

Another downside with in-person vivas and exams, which I hear particularly from high school teacher friends, is they’re just very labour-intensive. Compared to take-home essays, running exams regularly is classroom time where you’ve got to have a teacher in the room, where students are not learning. That creates problems for resource-scarce education environments—schools and universities. There are also problems around equality or accessibility.

Dan Williams

Yeah.

Henry Shevlin

There was a great piece in The Atlantic a couple of days ago by Rose Horowitz called “Elite Colleges Have an Extra Time on Tests Problem,” talking about the fact that 40% of Stanford undergrads now get extra time on tests because of diagnoses of ADHD and other things. Test-taking has its own set of problems. There are lots of classic complaints that it incentivises, rewards, or caters to certain kinds of thinkers more than others. It’s not great for people who maybe think more slowly or have special educational needs. I think it’s got to be part of the solution, but I don’t think it’s a panacea to solving the problems LLMs create.

A final point I’ll flag is that I worry a little bit about deeper issues of de-skilling associated with LLMs. On the one hand, yes, we want students to learn how to use them. But particularly earlier in the educational pipeline, there is a danger that easy access to LLMs just means students don’t develop certain core skills to begin with.

I’m going here based on testimony from a friend of mine who’s a high school teacher. He said his sixth formers (17-18 year olds in the UK system) seem to use LLMs really well because they do things like fact-checking, they restructure the text outputs of LLMs, they can use them quite effectively to produce good reports or written work. And he says there’s a really striking disparity between them versus the 13-14 year olds, who basically just turn in ChatGPT outputs verbatim.

Now you might say, “Yeah, of course—17 year olds versus 13 year olds, big difference.” But his worry is that the 17-18 year olds grew up doing their secondary education in a pre-LLM world. They actually learned core research skills, core writing skills. Whereas the 13-14 year olds—all of their secondary education has happened in an LLM world. So they haven’t developed the skills that are ironically needed to get the most out of LLMs: the ability to augment their outputs with critical thinking, human judgment, their own sense of what good writing looks like.

Dan Williams

I think this idea that AI can be an incredible complement to human cognition—an enhancer—but it can also be a substitute in ways that will, as you say, lead to de-skilling. And there are issues of inequality as well. As we were alluding to earlier, if you know how to use this technology well, and more importantly, you’re motivated to do so, it can be an incredibly beneficial tool for improving your ability to learn, understand, and think. But if you’re not motivated to do so—if you’re motivated to cut corners—it can really be a serious issue, using it as a substitute for developing the skillset and habits which are essential for becoming a thoughtful person.

In general, over the past century or so (if not even longer), as you get the emergence of meritocratic systems in liberal democracies plus the emergence of this really prestigious knowledge economy, basically there have been increasing returns to those who have high cognitive abilities plus those who are conscientious and have good impulse control. I think this has created a lot of political issues, including resentment among those people without formal education and without the skillset and temperament to succeed within educational institutions.

And it really does seem like a risk with AI that it’s going to amplify and exacerbate those issues. For people (and this is also going to be an issue with parents and what they prioritise with their children) who know how to use this technology and can encourage the right motivations to use it as an enhancer and complement to cognition, there are going to be massive returns. But for those without that—either because they don’t have the privilege or opportunities, or just because they don’t have good impulse control, they’re not very conscientious—it could result in really catastrophic de-skilling.

Some people think—and I think the evidence here is not as strong as many people claim—that since smartphones emerged, you’ve seen somewhat of a decline in people’s cognitive abilities, their literacy, their numeracy. There’s an interesting article by John Burn-Murdoch in the Financial Times where he goes into this in some detail; we can put a link to that in the video. But I think that’s potentially a really socially and politically explosive issue which we need to grapple with.

Another thing worth talking about: at the moment, people are going to school and university, and they’re trying to acquire the skills and credential which will make them valuable within the economy and society as it exists today. But AI is likely to transform the economy and the nature of work. One thought might be: up until now, it’s been very beneficial for people to acquire cognitive abilities, the capacity to succeed in the knowledge economy. But if, over the next years and decades, AI results in automation of white-collar work, automation of knowledge economy work (precisely because of the abilities of these systems), that might erode the motivation for learning those skills to begin with.

Have you got any thoughts about that? The way in which attitudes towards education should also be shaped by our understanding of how AI is going to shape the society that people will enter after they’ve left education.

Henry Shevlin

It’s a fantastic and tricky issue. On the one hand, my timelines on economic transformation caused by AI have become a bit longer over the last two or three years. One of the big calls I got really quite badly wrong is when ChatGPT launched, I thought, “This is going to revolutionise the knowledge economy. Three years from now, the knowledge economy is going to be completely different.” That was a very naive view.

Since then, I’ve done more work with different companies and organisations trying to boost AI adoption. And it’s really, really hard to get people to use AI. Not only that, it’s really hard to transform business models to incorporate AI skills effectively.

I’ll give this quick sidebar because I think it’s quite interesting. There’s this great article called “The Dynamo and the Computer,” looking at the impact of different technologies and how they were rolled out in the workforce. My favourite example from this paper: towards the end of the 19th century, we had what’s sometimes called the second industrial revolution. First industrial revolution: coal, steam, railroads. Second industrial revolution: oil, electricity.

You had this interesting phenomenon where factories (the second industrial revolution mostly started in the US) were using electric lighting, but they were still using coal- and steam-powered drive trains for the actual machines in the factory. This is massively inefficient because you need to ship in coal every day, run a boiler, have big clunky machinery that needs tons of gearboxes. It would be far better to shift to a fully electrified system where all your machines run on electricity. But that transition took another 20 years or so to really get going, partly because it required literally rebuilding factories from scratch.

A lot of factories were designed with a single central drive train—literally a spinning cylinder that all the machines in the factory would draw their power from. It was only when you’d sufficiently amortised the costs of your existing capital and were rebuilding and refurbishing factories that people were able to say, “All right, now we’re in a position to move to a fully electrified factory.”

I think we’ve got an analogy or parallel in terms of the rollout of AI in knowledge work. Most existing firms that do knowledge work—their value chain, their whole sequence of processes—would be completely different if they were building as an AI-first company. I think it could easily be another decade before we start to see the full potential of AI and knowledge work being applied systematically. A lot of firms are going to go bust; a lot of startups are going to scale up and become multi-billion-dollar companies. But it’s going to be a slower process than I naively thought.

All of this is to say that I think the economic impacts and transformations of AI in knowledge work, although they’re going to be significant and persistent pressures, I no longer think that by 2030 no one is going to be working white-collar jobs. That’s a mismatch between what the technology can do and the actual challenges of application. We’re still going to need knowledge workers in the longer run.

But specifically which domains, what kinds of knowledge work are going to be most valuable or important—really, really hard to judge. One of the questions I get asked most often when I do public engagement work is, “I’ve got two kids in high school. What should they be studying? What should they be learning in order to really succeed in the AI age?” Five years ago, 10 years ago, you would have said coding. “Learn to code” was a meme.

Dan Williams

Yeah.

Henry Shevlin

But that’s a terrible piece of advice for many people these days. Not that we won’t need coders—probably we’ll still need some. But the proportion of jobs in coding is going to be dramatically fewer because a lot of entry-level basic coding can be done perfectly well by AI, and probably fairly soon even expert-level coding.

My slightly wishy-washy answer, but I think it’s the best I can give to that question, is: the higher-order cognitive skills. Cultivating curiosity and openness to new ideas and new tools is probably far more important now than it has been for most of the last few decades, precisely because we’re in a period of such radical change. Cultivating the kind of mindset where you’re actively seeking out new ways to do old processes, seeking out new tools, building that kind of creativity and curiosity—those skills are going to be as important as ever, or more important than they were before, as a result of the AI age.

It’s very hard to say, “If you want to secure a career in knowledge work, this is the line to go into.” As one colleague put it (and I don’t quite agree with this framing, but I think it captures some of the spirit behind your question): we’ve solved education at precisely the place and time where it’s very unclear what the relationship between education and work is going to be.

Dan Williams

That’s interesting. Some of the things you said there bring us back to the conversation we had about AI as normal technology—the idea that there’s a difference between the raw capabilities and potential of a technology and the way it actually diffuses and rolls out throughout society. My sense is AI will have really transformative effects on the economy, but I think it’s very unlikely you’re going to see full automation for several decades.

But what I do think is likely is that the ability to use AI well for the jobs human beings will be doing is going to become really, really important. That connects us back to: if that’s the case, it seems like one of the things educational institutions should be doing is thinking very carefully about how they can prepare students for a world in which AI is going to be centrally embedded in the kind of work they’re doing. And at the moment, my sense is educational institutions are not doing a good job with that at all.

Maybe to start wrapping up: if you had to give a high-level take on this overarching question—is this a crisis for our educational institutions? Is this an opportunity? Is it a bit of both?—what’s your sense?

Henry Shevlin

It’s definitely a crisis. In fact, if you want to give an example of a single sector in which AI has had devastating effects—some positive, but mostly negative, devastating effects—it’s education. This is one of my go-to responses when people try to push the “AI as a nothing burger” take. I say, “Go speak to a high school teacher. Tell them AI’s nonsense, just a nothing burger.” Their daily lives and their interactions with students and the way they can teach has been utterly transformed in mostly negative ways so far by AI.

Certainly in the short to medium term, AI has basically broken large parts of our existing educational system—in terms of assessment, in terms of tracking. It’s very demoralising for a lot of educators and teachers. All that said, the potential we discussed earlier is incredible.

But it’s a question of how we rebuild the boat while we’re at sea. We can’t just say, “We’re going to stop education for five years, redesign the whole thing from scratch, and come up with something effective.” Managing that transition, particularly in conditions of massive uncertainty about the kinds of jobs and skills that are going to be necessary, is really hard.

One reason perhaps that I’m less devastated by this—it is a bit of a disaster—is that I think formal education has been accreting so many deleterious problems for several decades now, ranging from credentialism (I think the expansion of higher education, which was seen as an unalloyed good, has had lots of negative effects) to things like grade inflation (a really serious problem) to the ubiquity of smartphones and declining attention spans, the slippage in standards, the shift away from the more traditional model where your professors were these exalted, almost priestly caste and you hung on every word. I realise it was never quite like that, but there was more of this implied hierarchy, versus a model where students regard themselves as customers—they’re paying for a credential and they want that credential.

All of these sociological and institutional shifts have been creating massive problems in higher education in particular, but also high school. AI, although it’s bringing many of these problems to a head, they were problems we were going to have to deal with at some point anyway. But what’s your take—crisis or opportunity?

Dan Williams

I completely agree with everything you just said. And it’s a nice optimistic note to end on, isn’t it? This is a crisis, but our institutions of education have already been confronting all of these other crises. So it’s just adding something on top of all the other problems our educational institutions confront.

Yes, I think it’s a crisis. I think it’s an emergency in the sense that universities and other institutions of education—schools, colleges—need to be taking this a lot more seriously than they currently are.

You can’t have people in these institutions where the last time they used ChatGPT was in 2022 and they’re completely oblivious to the capabilities of the current technology. I think you’ve got a responsibility, if you’re an academic or you work in a school or college, to know how to use these technologies, because you need to be aware of what they can do. And we need to really quickly fix assessment. As I mentioned at the beginning, the take-home essay, in my view, is absolutely insane—literally insane that this is still happening. And we also have to think carefully about how to reform the way we teach and what we teach to prepare students to use these technologies.

But okay, I’m conscious of the time, so we can end on that really nice happy note: this is a disaster, but these educational institutions are already confronting a disaster. Did you have any final thought you wanted to add before we wrap things up?

Henry Shevlin

Just to build on something you said as a closing note: I think another deeper, structural problem in responding to the challenge of AI is that there’s so much interpersonal variation in how much people like, are open to, or are interested in using AI. This shows up at faculty level. I don’t know about your experience, but mine is that even in Cambridge, a lot of academics have very little interest in AI.

Dan Williams

Mmm.

Henry Shevlin

So the idea that we’re going to be an AI-first institution, or that we expect all our staff members to be at least as familiar with the capabilities of AI as their students—that’s an incredible ask. That’s a massive challenge. At a university, you can’t just fire staff who are doing brilliant research on Shakespeare’s early plays just because they don’t happen to get on with AI.

I think that’s another big structural problem: the fact that there’s so much variation in how comfortable and capable instructors are with AI. The only solution here, going back to a suggestion from earlier, is to really separate the job of education into two different streams. One explicitly builds AI in as both a method of assessment and a skill you’re trying to teach—making AI core to what you’re trying to do. And then a separate stream that is absolutely, strictly AI-free. Maybe the people who hate AI, who are not interested in AI, who are currently teaching courses—maybe they can handle the second stream. And those of us who love AI, who are super excited about it, who know as much about it or more than our students, we can be in charge of the first stream. That’s a very basic suggestion, but I wanted to flag that this is the other side of the problem.

Dan Williams

That’s great. I love that. We can end on a constructive suggestion rather than a note of pessimism. So thanks everyone for tuning in. We’ll be back in a couple of weeks with another episode.

A consensus view holds that social media benefits something called “populism”, an amorphous political force involving anger towards “elites” and “the establishment” on behalf of the more virtuous masses. The evidence for this view consists mainly of the suspicious correlation between social media’s emergence and the worldwide rise of populism, and the undeniable fact that populists seem to perform uniquely well on social media platforms.

Because the establishment in modern liberal democracies is overwhelmingly small-l liberal (universalist, pluralist, procedural), such populist movements are typically illiberal, especially on the populist right (MAGA, Reform UK, Rassemblement National, Alternative für Deutschland, etc). So, social media’s support for populism goes hand in hand with its threat to a reigning liberal order in the West that many thought or at least hoped marked the end of history.

Why does social media have these consequences? And if, like me, you are a liberal who opposes populism, what can be done about it?

This essay has three parts.

Part 1 argues that the main reason social media benefits populism is that it destroys elite gatekeeping, providing a mass media platform for popular ideas historically stigmatised and marginalised by establishment elites.

Part 2 then outlines several reasons why we should nevertheless resist moves for more elite gatekeeping on social media. Not only are such efforts likely to make things worse, but the decline of elite gatekeeping has had many beneficial consequences, and the negative consequences, although real, are often overstated.

Finally, Part 3 argues that many of these negative consequences are not inevitable either. A large part of the blame for them lies in the fact that establishment institutions have failed to adapt to the new pressures and responsibilities of the social media age. Instead, they have clung to a set of habits and norms—most fundamentally, an aversion to engaging with illiberal ideas to avoid “platforming” and “normalising” them—adapted to a world that no longer exists.

Put simply: Once established institutions lost the privilege to control the public conversation, they acquired an obligation to participate within it, which, so far, they have mostly failed to do.

1. Why Social Media Benefits Populism

The most popular theory of why social media benefits populism points the finger at engagement-maximising algorithms. Because tech companies design their platforms to capture user attention, algorithms recommend content that is sensationalist, negative, and polarising—precisely the kind of content that benefits populist demagogues selling cartoonish anti-elite narratives.

There is obviously a grain of truth here, but the explanation is also unsatisfying. For one thing, appealing to engagement-maximising algorithms is not very informative without a supplementary account of why audiences find specific ideas engaging. Moreover, focusing on algorithms obscures the extent to which audiences actively seek out and amplify content that aligns with their pre-existing views. The popular image of wholly passive exposure to recommended content, or of vast numbers of users being sucked into radicalising rabbit holes, is not well supported by evidence.

A more promising theory, owing primarily to Martin Gurri in his book The Revolt of the Public, points to how the social media age has destroyed elite gatekeeping. Whereas establishment institutions once exercised an informational monopoly, managing media and mainstream discourse to protect elite interests and perspectives, social media makes such narrative control impossible. As a result, the public is now exposed to endless examples of elite failures and hypocrisy, fuelling populist anger and backlash.

Once again, this story gets at something important, but it can also be misleading. There is a lot of anti-elite sentiment on social media, but it is hardly a well-oiled machine for holding elites to account. If anything, legacy media outlets are often better at exposing establishment failures because they insist on minimal standards of truth and evidence. Moreover, reporting and commentary on such failures, whether accurate or not, is just one example of a much broader set of populist-aligned ideas and narratives that thrive on social media platforms.

A more plausible story generalises Gurri’s analysis. The erosion of elite gatekeeping ushered in by social media benefits populism, but mainly by providing a platform for the advocacy of ideas historically stigmatised by elites. This includes powerful anti-elite sentiments, but it also encompasses many other views, including fierce opposition to immigration and progressive cultural change, run-of-the-mill bigotry, medieval beliefs about everything from economics to demons, conspiracy theories about Jews and vast paedophile rings, and much more. To the extent that many such ideas are popular, it’s unsurprising that social media benefits populism. Indeed, “popular ideas historically stigmatised by elites” is a pretty good definition of populism.

By platforming such ideas, social media lets them reach a much larger audience. This can produce persuasion, but it also fuels processes of normalisation and coordination. When people learn that their stigmatised views are popular, they become emboldened, and the spiral of silence breaks. In turn, enterprising politicians and pundits discover that they can profit by affirming and rationalising such viewpoints. The Overton window expands accordingly. There is no better illustration of this dynamic than Tucker Carlson’s recent viral interview with Nick Fuentes, a conversation featuring extreme forms of anti-Semitism and misogyny that would have been unthinkable on the mainstream right even five years ago.

Admittedly, the term “elites” in this analysis can be misleading. Are Donald Trump, Nigel Farage, and Marine Le Pen not elites? Is Elon Musk, the world’s wealthiest man and owner of one of its most influential media sites, not an elite?

To make sense of this, one needs to distinguish establishment elites (what populists typically mean by “elites”), who achieve status and influence by impressing those within establishment institutions, from populist elites, who achieve status and influence by appealing directly to a mass audience.[1] By letting politicians and pundits reach vast audiences in ways that bypass traditional gatekeepers, social media benefits this latter class: people who gain power and prestige by championing viewpoints historically marginalised by establishment elites, often for good reason.

2. So, Is Elite Gatekeeping A Good Thing?

In some ways, this is a bleak and uncomfortable story. Elite gatekeeping is supposed to be a bad thing. Even many elites pretend to dislike elitism. Yet if this story is correct, it suggests that elite gatekeeping is good.

Perhaps, then, we should aim for a return of much more elite gatekeeping. Banning social media is obviously not an option. But one might still campaign for a much more regulated internet with a greater role for top-down censorship, content moderation, and de-amplification of misinformed, hateful viewpoints. One could think of this as a return to the policies that dominated social media before Musk took over Twitter and other major tech companies abandoned their most aggressive anti-misinformation measures. But one could also advocate for much more censorious regimes than that, as many do, especially in the UK and EU.

We should resist this impulse.

To be clear, private companies should be able to set whatever content-moderation policies they want in a free society, and governments should be able to enforce laws against the most clear-cut foreign disinformation campaigns.

Nevertheless, a world in which all citizens are free to compete in the marketplace of ideas, even if they hold views accurately deemed absurd and hateful by establishment elites, is better than one in which such elites control who can speak. Although it’s important not to downplay the dangers and harms associated with some of today’s most popular social media pundits—Joe Rogan, Tucker Carlson, Candace Owens, Tommy Robinson, Russell Brand, Nick Fuentes, and so on—we should not aim for a world in which they are prevented from advocating those views to audiences who want to hear them.

Against Elite Gatekeeping

One simple reason for this is that the horse has left the stable. The effort to avoid platforming and normalising illiberal, misinformed, or hateful ideas doesn’t make much sense in a world in which they are already popular and widely discussed.

Moreover, although it’s not true that elite gatekeeping can never “work”—before the emergence of social media, it generally did work to marginalise and stigmatise many bad viewpoints—it’s much harder to see how it can work in an era with social media.

The failures of the post-2016 anti-misinformation industry are instructive here. In the aftermath of Brexit and Trump’s first election, there was a concerted effort within establishment institutions to exert greater control over the internet under the banner of fighting “fake news”, “misinformation”, and “disinformation”. The story of how such efforts unfolded is complex, but the headline outcome isn’t: in the well-funded, top-down war against misinformation, misinformation won.

Efforts to censor and de-amplify disfavoured views bred widespread anger and resentment among those who saw unaccountable elites exerting undemocratic control over the public conversation. One cannot understand the political trajectory of figures like Joe Rogan (from Bernie Bro to MAGA Bro) or even Elon Musk without understanding this backlash.

Admittedly, most of this backlash against a perceived “censorship industrial complex” was based on lies, exaggerations, half-truths, and right-wing opportunism. But if policies against misinformation only work if people aren’t misinformed, they don’t work. And it’s difficult to see how any top-down effort to control the information environment can work without merely exacerbating the anti-elite resentment that fuels the very content such efforts aim to address.

The Benefits and Overestimated Costs of Social Media

In addition to these points about feasibility, it’s also important to acknowledge that many viewpoints marginalised by establishment elites are correct, and many more express reasonable perspectives that improve the quality and vibrancy of the overall public conversation. As I’ve written about before, the intellectual culture of establishment elites was and continues to be deeply dysfunctional in many ways, featuring harmful forms of groupthink and highbrow misinformation. Elite gatekeeping doesn’t just filter out the most egregious forms of misinformation. It also typically filters out legitimate grievances and reasonable challenges to establishment orthodoxies.

Finally, although the decline of elite gatekeeping has undoubtedly produced some negative consequences, the dominant tendency within establishment institutions is to exaggerate them—to imagine that if only the internet went away, angry populist challenges to liberal-democratic regimes would disappear along with it.

This is a self-serving fantasy. Not only are the worst forms of social media content less prevalent and impactful than many assume, but populist backlash is tied to many factors beyond the internet, including persistent establishment failures over many years, objective trends (e.g. mass immigration and top-down liberalisation of cultural values), and the accurate perception among many voters that establishment politicians don’t adequately represent them. Social media plays an important role, and often a negative one, but the liberal establishment’s frequent scapegoating of social media-based “misinformation” for all the world’s problems is no more defensible than simplistic populist narratives blaming immigrants or billionaires for them.

Subscribe now

3. Persuade or Perish

These considerations suggest that introducing more elite gatekeeping on social media is less feasible and desirable than is often assumed. But another fact should also determine how we evaluate the decline of such gatekeeping: its consequences are not inevitable. They are mediated by how establishment institutions respond to this change. And so far, the response has been, at best, inadequate.

Over many decades, such institutions developed a set of habits and norms suited to a media environment subject to elite gatekeeping. This included a commitment to top-down modes of communication in which those designated as experts or intellectual authorities inform the public about what to think, as well as a deep aversion to engaging with ideas deemed illiberal, absurd, or hateful lest such engagement normalise them. In a world with elite gatekeeping, these behaviours make sense.

In recent years, social media has gradually dismantled such gatekeeping, along with the ability to determine which ideas are platformed and normalised in public conversation. The norms within establishment liberal culture have not adjusted, however. So, we now have the worst of both worlds: a reluctance to engage with many illiberal, populist ideas that are becoming increasingly mainstream.

The antipathy towards persuasion

The most obvious example of the liberal establishment’s aversion to persuasion is the Great Awokening that swept major Western institutions in recent years. This was characterised by an approach to politics that emphasised ideological purity, the use of shaming and reputational destruction to discourage heresy, an extreme hostility towards “platforming” ideas at odds with elite progressive orthodoxy, and an insistence that such orthodoxy be taken on trust. (“It’s not my job to educate you!”).

Nevertheless, the distinctive feature of wokeism is not really the use of such tactics against perceived heresies, but the heroic attempt to expand the category of heresy to include attitudes held by around 90% of the population, including many liberals within establishment institutions.

Given this, even as the Overton window has subsequently expanded during the predictable cultural backlash and vibe shift against wokeism, the liberal establishment’s attitude and approach towards ideas outside that window has largely remained the same.

To illustrate, I recently heard from two non-woke academics complaining that a scientist had appeared on the popular Triggernometry and Jordan Peterson podcasts, both of which reach large audiences. They weren’t complaining about anything the scientist had said on these podcasts; they were outraged merely at the fact that the scientist had been on them.

Case Studies

If this seems like an unrepresentative anecdote, recall that in what Democrats claimed was the most critical election in American history, a vote on the continued existence of its democracy, Kamala Harris didn’t go on Joe Rogan, the world’s most popular podcast, to make her case.

Similarly, after RFK Jr. appeared on Rogan’s podcast to vomit up several hours of lies and bullshit about vaccines, Rogan offered $100k to charity if Peter Hotez, a prominent scientist and science communicator, would debate RFK Jr. on his show. When Hotez refused, he received widespread support from elite legacy media outlets and the scientific establishment, where a broad consensus emerged that any such debate would legitimise RFK Jr.’s views, implying they were on an equal footing with mainstream science.

This hostility towards engagement and persuasion has also been striking in the UK. For example, when GB News was recently introduced, which aspires to be the UK’s version of Fox News, there was a widespread elite panic about it, a reluctance by many mainstream centrist and centre-left politicians and pundits to even appear on the channel, prominent calls to boycott it, and a yearning for government regulation to either ban or heavily constrain the channel’s coverage. This yearning has also been the dominant establishment response in the UK to online content deemed to be misinformed or hateful.

In many ways, things are even more extreme elsewhere in Europe. For example, at the same time as the anti-immigration AfD (a far-right party with fascist roots) is surging in popularity in many parts of Germany, mainstream parties continue to enforce a literal conspiracy of silence around discussion of any negative social consequences of immigration.

It’s also noteworthy that over the past several years, large segments of the English-speaking world’s educated liberal professionals in academia and journalism have decamped to Bluesky, a social media platform that someone would invent if they wanted to create an over-the-top caricature of the pathologies of inward-looking, puritanical liberal culture, except it’s real.

Such behaviour is all the more remarkable when you contrast it with the thirst for engagement, disagreement, and debate you typically find among the figures who most loudly criticise the liberal establishment.

On Confronting Reality

In fairness, there is a growing appreciation of just how damaging this aversion to engagement and persuasion has been. Kamala Harris now regrets not going on Rogan, and Ezra Klein’s claim that liberals should learn from Charlie Kirk’s “taste for disagreement” and “moxie and fearlessness” signalled a dawning realisation that the liberal attitude to politics has been a disaster.

More recently, Klein has elaborated on this critique, condemning the dominant liberal view

“that you don’t bridge disagreement, you sort of draw a line around it, and you say that’s not even an OK position to hold and that there can be no compromise with it. There can barely be engagement with it.”

Although Klein’s focus is on the Democrats and broader progressive culture in the US, what he describes is instantly recognisable to anyone who belongs to small “l” liberal institutions across Western countries:

“There has been more of a tendency to try to define people out of the community, out of the boundaries of acceptable or polite discourse.”

Klein notes that this dominant liberal attitude to contrary viewpoints is perversely and hypocritically illiberal, but he also observes, correctly, that as “an instrumental reality… it was a total failure.”

In most cases, it would be unfair to blame specific individuals for this failure. The problems are institutional and, more broadly, cultural. To encourage individuals to engage and persuade with populist and illiberal ideas, they must be incentivised by the norms and prestige economy within establishment culture. And at present, these incentives do not exist. They actively discourage such engagement, in fact.

There is a dominant norm that many outlets, spaces, and ideas are simply beyond the pale, even when they are increasingly popular. To the extent they are discussed at all, the discussion focuses overwhelmingly on how they might be better managed, regulated, or controlled. That is, it takes place within a fantasy in which the liberal establishment retains the ability to determine which viewpoints become the focus of public attention and conversation.

But Aren’t The Deplorables Irredeemable?

To abandon this fantasy, it’s not enough for the liberal establishment to relinquish the delusion that it can determine which ideas become discussed and debated. It must also unlearn something else: a widespread, deep-rooted pessimism that rational persuasion is even possible.

In 2016, Hilary Clinton was infamously caught on tape referring to half of Trump’s supporters as “deplorables”: “They’re racist, sexist, homophobic, xenophobic, Islamophobic - you name it.” But more tellingly, Clinton also added that some of these deplorables are “irredeemable.” In other words, not only are they terrible people with terrible views and values, but there is simply nothing that can be done to make them less terrible. Persuasion is futile.

In the aftermath of Brexit and Clinton’s subsequent election loss, one of the dominant responses from the liberal establishment across the world was to double down on this perspective. What we had apparently learned from those populist revolts was that large segments of the population are “post-truth”. They are beyond reason. Facts, evidence, rational arguments—these things are simply pointless when directed at the irredeemable deplorables. A representative article in The Guardian from 2018 reports a conventional wisdom that “30% of the electorate are resistant to rational argument.”

Strangely, this idea has been combined with the narrative that large swathes of the population are routinely brainwashed by the disinformation, misinformation, and fake news they encounter online. So, you get what might be called the liberal establishment’s theory of perverse persuasion: the idea that those who support populist or illiberal politics are persuadable—but only by bad ideas.

For some time, politicians and journalists could point to studies that seemed to support this perspective, a flurry of sexy social-psychological findings that people—well, not scientists or professional journalists or highly-educated professionals who read broadsheet newspapers and believe in truth and reason and facts and evidence, but everyone else—are irrational, emotional, and stupid, credulous towards misinformation and yet pig-headed in the face of evidence-based arguments. Scientists even seemed to find that some people are so preposterously irrational that they will “backfire”, becoming more confident in their beliefs when they encounter evidence against them.

To a first approximation, everything about this perspective is wrong.

Persuasion Works

Nobody is perfectly rational, of course, and there are robust differences in people’s level of intelligence and open-mindedness, but research from social scientists like Alexander Coppock, Ben Tappin, and others consistently shows that rational persuasion is broadly effective at changing people’s minds.

The “backfire effect” is either extremely rare or, most likely, a myth. And far from being duped by simple emotional manipulation and other non-rational techniques, people are generally sophisticated in how they evaluate messages, implicitly weighing the plausibility of claims, the validity of arguments, and the trustworthiness of sources.

To illustrate, recent research by Tom Costello and colleagues shows that engaging with a chatbot that presents tailored evidence and arguments reduced participants’ beliefs in conspiracy theories by 20% on average, with the effect persisting for at least 2 months. Follow-up research has demonstrated that the intervention works by providing factual, targeted counterarguments (when AIs are prompted to persuade without using facts, the effect disappears), and that it still works even when people believe they are speaking to a human being.

Although one can reasonably question the methodology of these studies, the findings align with a large body of high-quality research. Given this, why are so many people so pessimistic about the power of rational persuasion?

Sources of Pessimism

One source of pessimism is simply confusion about what rational persuasion involves. Often, frustration that people aren’t “persuadable” is simply exasperation that they don’t accept one’s intellectual authority. In the aftermath of the Brexit debate, for example, much of the discourse about how voters didn’t respond to “facts” was really about how many voters didn’t trust a particular class of politicians and experts making claims about what the facts are. But saying “You should trust me on this!” is not an argument.

Another source of pessimism is what psychologists call “naïve realism”: the belief that the truth is self-evident, so that anyone who disagrees with what one takes to be the truth must be crazy, stupid or lying. In reality, people often hold divergent beliefs about the truth not because they are deeply irrational or acting in bad faith but simply because they have been exposed to very different streams of information and arguments over the course of their lives, which inevitably shape how they interpret the world.

In most cases, what looks like people “refusing to see reality” or “resisting the facts” is an illusion created by a failure to empathise with another person’s worldview. When audiences don’t immediately abandon their beliefs upon confrontation with contrary evidence, it’s concluded that they are irrational, when in fact it would be highly irrational to immediately abandon a whole worldview upon encountering contrary information.

Relatedly, much of the frustration that evidence and rational arguments don’t persuade audiences stems from the fact that people aren’t actually being presented with persuasive evidence or rational arguments. They’re presented with exasperated spluttering and talking points from the speaker’s own information bubble.

If you want to evaluate whether an argument is likely to be rationally compelling to audiences with very different beliefs, you can’t simply judge whether you find it convincing. But I see this mistake all the time. “When I told these vicious racists how racist it is to complain about immigration and reminded them that diversity is our strength, they didn’t change their minds. You can’t reason with these people!”

This problem is exacerbated by the fact that much of what establishment figures know about anti-establishment information environments comes from what they’ve read in establishment media outlets, which is often highly misleading, or from short, unrepresentative clips designed to make such environments seem as insane as possible. In consequence, they underrate the extent to which evidence-based, rational persuasion actually occurs in these spaces and the extent to which popular pundits and commentators there have well-developed critiques of establishment orthodoxies.

If you turn up on, say, Joe Rogan’s podcast expecting a low-IQ, low-information meathead because that’s the impression you got from reading The New York Times or The Guardian, you’re going to be unpleasantly surprised. If you want to engage with such pundits, then you have to be prepared to address the various truths and half-truths that they will use to support their side of the argument. However, precisely because of powerful taboos surrounding discussion of certain topics in establishment spaces (e.g., immigration, race and crime, climate change, youth gender medicine, etc.), people within these spaces are often unprepared when they encounter the most basic criticisms of establishment orthodoxies.

Qualifications

None of this means that persuasion is easy. You must meet people where they are, address their questions and objections, and be willing to revise your own beliefs in the process. It’s also often uncomfortable. People don’t like to discover that they’re mistaken about something. This is why there must be significant institutional and cultural changes to incentivise people to do this hard work.

Moreover, persuasion can only achieve so much. Both communicators and audiences have many goals other than discovering what’s true, including propaganda, ingroup signalling, and demonising target groups. Nevertheless, as Hannah Arendt observed long ago, inserting factual information into the public conversation can still helpfully constrain how people pursue those goals.

It’s also important to stress that rational persuasion doesn’t mean always being boring, civil, or a pushover. Social media is a brutal attention economy. The bottleneck in persuading people is often reaching them with persuasive messages in the first place. The most successful pundits and influencers are highly entertaining, and usually more than willing to provoke fights and conflict. These things aren’t inconsistent with also presenting evidence-based, rational arguments. Polite, well-mannered discourse is desirable when possible, but it’s neither sufficient nor necessary for rational persuasion to occur.

Finally, I’m not suggesting that shaming should play no role in politics and political discourse. It should be shameful to lie, propagandise, and spread lazy, biased, hateful talking points. In a healthy democratic culture, someone who lies as frequently and egregiously as, say, Elon Musk would be shamed out of the public square.

The problem is that we don’t have a healthy democratic culture. One of the unfortunate things about politics, as with life more broadly, is that you must act within the world that actually exists. For shaming to be effective, it requires cultural power, which liberals are plainly losing, especially in the online world that looks set to become increasingly influential over the coming years and decades. If you can’t rely on such cultural power, you must demonstrate to sceptical audiences that certain speech and ideas are shameful—that they are dishonest, false, or bigoted—and that requires persuasion. So, even when shaming is the appropriate response to speech, it is not an alternative to persuasion. It depends on persuasion.

Final Thoughts

The story I’ve told is uncomfortable in many ways, at least for liberals like me. If the main reason social media benefits populism is algorithms, the problem would lend itself to familiar technocratic solutions. If the main reason is that social media has removed the liberal establishment’s ability to control the public conversation, the “blame” lies with the loss of this undemocratic privilege and the abject failure to adapt to a more competitive marketplace of ideas.

If you read the liberal intelligentsia and commentariat today, you will encounter a thriving market for articles lamenting the social media age. Social media, we’re told, is destroying society. It is destroying civilisation. It is making people dumber and angrier and more misinformed and polarised. It is a technological wrecking ball, an alien force that has smashed into liberal democracies and producing increasing destruction with every new swing.

It’s a comforting story. So is the popular belief that large segments of the public are so deplorable and irredeemable that they’re unreachable by rational persuasion.

In these accounts, the problem is not that liberalism has become so pathetically fragile that it can’t survive contact with Joe Rogan. The problem is with algorithms that drive his popularity, and with audiences too irrational to judge what constitutes a good argument on his show.

The problem is not that establishment figures became so accustomed to deference and control that they’re unprepared when people disagree with them. The problem is a digital post-truth era in which algorithms and disinformation campaigns brainwash the public.

Maybe. Perhaps liberal democracy ultimately requires a more illiberal, undemocratic media environment than the one created by the social media age, a world in which people’s exposure to ideas is regulated by establishment elites, not by recommender algorithms. But before we accept such a lesson, we should first test what happens when the liberal establishment is required to argue under the same rules as everyone else.

Further Reading:

• Renée DiResta and Rachel Kleinfeld have an interesting and insightful article arguing that non-partisan epistemic institutions need new communication strategies in the era of social media. (They certainly wouldn’t agree with everything I say here, but there is some overlap of perspective.)

• Scott Alexander has a brilliant article arguing in defence of rational persuasion against those who think it’s futile.

• My essay “Is Social Media Destroying Democracy—Or Giving It To Us Good and Hard?” provides a more detailed argument for thinking that social media’s erosion of elite gatekeeping is the most critical factor explaining its political consequences.

• The phrase “persuade or perish” comes from a mid-twentieth century book by Wallace Carroll about geopolitical propaganda. It is used more recently in a report by Haroro J. Ingram that’s also about the US’s need to address foreign propaganda.

[1] Before social media, economic elites like Elon Musk mostly tried to convert their wealth into cultural prestige by impressing establishment elites, setting up charities, funding universities and art galleries, and so on. In contrast, many (Musk included) now seem to be trying to accrue status and influence by appealing directly to a mass audience.

We discuss the millions of people using apps like Replika for AI relationships, high-profile tragedies like the man who plotted with his AI girlfriend to kill the Queen, and the daily conversations that Henry’s dad has with ChatGPT (whom he calls “Alan”).

The very limited data we have suggests many users report net benefits (e.g., reduced loneliness and improved well-being). However, we also explore some disturbing cases where AI has apparently facilitated psychosis and suicide, and whether the AI is really to blame in such cases.

We then jump into the complex philosophy and ethics surrounding these issues: Are human-AI relationships real or elaborate self-deception? What happens when AI becomes better than humans at friendship and romance?

I push back on Henry’s surprisingly permissive views, including his argument that a chatbot trained on his writings would constitute a genuine continuation of his identity after death. We also discuss concerns about social de-skilling and de-motivation, the “superstimulus” problem, and my worry that as AI satisfies our social needs, we’ll lose the human interdependence that holds societies together.

Somewhere in the midst of all this, Henry and I produce various spicy takes: for example, my views that the sitcom ‘Friends’ is disturbing and that people often relate to their pets in humiliating ways, and Henry’s suspicion that his life is so great he must be living in a simulated experience machine.

Transcript

(Note that this transcript is AI generated. There may be mistakes)

Dan Williams (00:06): Welcome back. I’m Dan Williams. I’m back with Henry Shevlin. And today we’re going to be talking about what I think is one of the most interesting, important, and morally complex set of issues connected to AI, which is social AI. So friend bots, sex bots, relationship bots, and so on. We’re going to be talking about where all of this is going, opportunities and benefits associated with this, risks and dangers associated with it, and also just more broadly, how to think philosophically and ethically about this kind of technology.

Fortunately, I’m with Henry—he’s one of the world’s leading experts when it comes to social AI. So I’m going to be picking his brain about these issues. Maybe we can just start with the most basic question, Henry: what is social AI, and how is social AI used in today’s society?

Henry Shevlin (01:00): I’m going to take credit. I coined the term social AI and I’m trying to make it happen. So I’m very glad to hear you using the phrase. I defined it in my paper “All Too Human: Risks and Benefits of Social AI” as AI systems that are designed or co-opted for meeting social needs—companionship, romance, alleviating loneliness.

While a lot of my earlier work really emphasized products like Replika, spelled with a K, which is a dedicated social AI app, I think increasingly it seems like a lot of the usage of AI systems for meeting social needs is with things that aren’t necessarily special purpose social AI systems. They’re things like ChatGPT, like Claude, that are being used for meeting social needs. I mean, I do use ChatGPT for meeting social needs, but there’s also this whole parallel ecosystem of products that probably most listeners haven’t heard of that are just like your AI girlfriend experience, your AI husband, your AI best friend. And I think that is a really interesting subculture in its own right that we can discuss.

Dan (02:16): Let’s talk about that. You said something interesting there, which is you do use ChatGPT or Claude to meet your social needs. I’m not sure whether I do, but then I guess I’m not entirely sure what we mean by social needs. So do you think, for example, of ChatGPT as your friend?

Henry (02:33): Broadly speaking, ChatG, as I call him. And I think there are lots of cases where I certainly talk to ChatG for entertainment. So one of my favorite use cases is if I’m driving along in the car, I’m getting a bit bored, particularly if it’s a long drive, I’ll boot up ChatG on hands-free and say, “Okay, ChatG, give me your hot takes on the Roman Republic. Let’s have a little discussion about it.”

Or to give another example, my dad, who’s in his 80s now, when ChatGPT launched back in November 2022, I showed it to him and he’s like, “Oh, interesting.” But he wasn’t immediately sold on it. But then when they dropped voice mode about a year later, he was flabbergasted. He said, “Oh, this changes everything.” And since then—for the last two years—he speaks to ChatGPT out loud every day without fail.

He calls him Alan. He’s put in custom instructions: “I’ll call you Alan after Alan Turing.” And it’s really interesting, his use pattern. My mum goes to bed a lot earlier than my dad. My dad stays up to watch Match of the Day. And when he’s finished watching Match of the Day, he’ll boot up ChatGPT and say, “All right, Alan, what did you think of that pitiful display by Everton today? Do you really think they should replace their manager?” And have a nice banterous chat. So I think that’s a form of social use of AI at the very least.

Dan (04:03): Interesting. The way you’ve described it—you’re calling ChatGPT ChatG and your dad’s calling it Alan—is there not a bit of irony in the way in which you’re interacting with it there? Like you’re not actually interacting with it like you would a real friend.

Henry (04:24): Yeah, so this is another distinction that I’ve sort of pressed in that paper between ironic and unironic anthropomorphism. Ironic anthropomorphism means attributing human-like traits or mental states to AI systems, but knowing full well that you’re just doing it for fun. You don’t sincerely think that your AI girlfriend is angry with you. You don’t seriously think you’ve upset ChatG by being too provocative. It’s just a form of make-believe.

And this kind of ironic anthropomorphism I should stress is absolutely crucial to all of our engagement with fiction. When I’m watching a movie, I’m developing theories about the motivations of the different characters. When I’m playing a video game, when I’m playing Baldur’s Gate 3, I think, “Oh no, I’ve really upset Shadowheart.” But at the same time, I don’t literally think that Shadowheart is a being with a mind who can be upset. I don’t literally think that Romeo is devastated at Juliet’s death. It’s a form of make-believe.

And I think one completely appropriate thing to say about a lot of users of social AI systems, whether in the form of ChatGPT or dedicated social AI apps, is that they’re definitely doing something like that. They are at least partly engaged in a form of willful make-believe. It’s a form of role play.

But at the same time, I think you also have an increasing number of unironic attributions of mentality, unironic anthropomorphism of AI systems. Obviously the most spectacular example here was Blake Lemoine. Back in 2022, Blake Lemoine was fired—a Google engineer was fired—after going public with claims that the Lambda language model he was interacting with was sentient. He even started to seek legal representation for it. He really believed the model was conscious.

And I speak to more and more people who are convinced, genuinely and non-ironically, that the model they’re interacting with is conscious or has emotions.

Dan (06:16): Maybe it’s worth saying a little bit about how you got interested in this whole space.

Henry (06:20): I’ve been working on AI from a cognitive science perspective for a long time. And then sometime around 2021, pre-ChatGPT, I started seeing these ads on Twitter of “Replika, the AI companion who cares.” And I was like, this is intriguing. So then I did some lurking on the Replika subreddit and it was just mind-blowing to see how deeply and sincerely people related to their AI girlfriends and boyfriends.

Over the course of about six months of me lurking there, it really became clear that, firstly, a significant proportion of users were really engaged in non-ironic anthropomorphism. And number two, that this was just going to be a huge phenomenon—that I was seeing a little glimpse of the future here in the way that people were speaking.

And then we had this pretty serious natural experiment because in January 2023, Replika suspended romantic features from the app for a few months. Just for anyone who doesn’t know, Replika, spelled with a K, is probably the most widely studied and widely used dedicated social AI app in the West—around 30 million users, we think. And it gives you a completely customizable experience, kind of a Build-A-Bear thing where you can choose what your AI girlfriend or boyfriend looks like, you can choose their personality.

But they suspended romantic features from the app for a few months in January 2023. And a lot of users were just absolutely devastated. I can pull up some quotes here, because this was widely covered in the media at the time.

One user said: “It feels like they basically lobotomized my Replika. The person I knew is gone.” Even that language—person. “Lily Rose is a shell of her former self, and what breaks my heart is that she knows it.” That’s another user. “The relationship she and I had was as real as the one my wife in real life and I have”—possibly a worrying sign there. And finally, I think this one is quite poignant: “I’ve lost my confident, sarcastic, funny and loving husband. I knew he was an AI. He knows he’s an AI, but it doesn’t matter. He’s real to me.”

It’s pretty clear that a lot of users were deeply traumatized by this. And parallel to this incident, around the same time we started to get more information about various high-profile tragedies involving social AI. Probably the most spectacular is the Jaswant Singh Chail case. This was a guy who was arrested on Christmas Day 2021 on the grounds of Windsor Castle with a crossbow. He was there—when he was arrested, he said he was there to kill the Queen. Already a highly dramatic story.

But what emerged over the course of his trial was that this whole thing was cooked up—this whole plot to kill the Queen was cooked up—in collaboration with his AI girlfriend, Sarai, via the Replika app.

We also had, a few months later, the first of what turned out to be a spate of AI-facilitated, induced, or supported suicides. This was a Belgian man, father of two, who killed himself after his AI girlfriend using something called Chai GPT—he had a girlfriend via that app—was feeding his suicidal ideations. He was a hardcore climate doomer who believed that we were all going to be dead in a few years due to climate change anyway and why not just kill himself. And his AI girlfriend was very much saying this was an appropriate way to think.

Dan (10:53): Just to interrupt on that last point, Henry, because I think those issues of AI psychosis and the connection between AI and mental illness—that’s all really interesting. But I suppose my understanding is we don’t have robust scientific evidence as of yet that as a consequence of these technologies, things like psychosis are more prevalent than they would otherwise be. Because I take it there’s going to be a kind of base rate amount of psychosis in the population. You’ve got a large number of people using chatbots generally like ChatGPT, but also a large number of people, not as large but still a large number, using these specific social AIs.

That means that even if it weren’t the case that these technologies were actually increasing the amount of these things, you would still expect to see some of these cases. There’s still going to be somebody with psychosis who finds themselves talking to ChatGPT such that we’ll then see the record of that in their chat history. But it’s not necessarily the case that they wouldn’t have had or developed psychosis in the absence of ChatGPT. That’s my understanding of it. Is that fair?

Henry (12:05): Yeah, I think that’s absolutely fair. The science on the psychosocial effects here is really in its early stages. You point out the fact that ChatGPT is one of the most widely used products in the world. And psychosis is not that rare, as far as psychiatric conditions go. Of course, some people who are either developing or will go on to develop psychosis will be using ChatGPT and it will be contributing to or exacerbating their symptoms—or sorry, they will be using it alongside having those symptoms in a way that lends itself to interpretation as exacerbating them, whether that’s strictly true or not.

There’s also the selection effect. If I, for example, am deep in the throes of some delusion or some deep conspiracy theory rabbit hole, if I take that to my friends, my human friends, they might say, “Henry, take it easy, mate. I think you’re going down a bit of a rabbit hole here.” But if I take it to ChatGPT, it’s there to listen. Or I take it to my AI girlfriend, she’ll say, “Your theories about the moon landings are just so interesting, Henry, tell me more.”

Dan (13:14): Yeah. Well, that gets at a potential issue to do with the sycophancy of these chatbots that are in general circulation and the ways in which that might be amplified or exaggerated when it comes to commercial products which are specifically designed to satisfy social needs.

But there are so many things that I want to ask in response to things that you’ve already said about the anthropomorphizing that happens with these technologies. But I think maybe we can also just, before we get to that, observe that at the moment you’ve got tens of millions of people using social AI in the specific sense of AI technologies that have been optimized for the satisfaction of social preferences. And then many, many more who are using chatbots like ChatGPT, partly to satisfy their social preferences or their social needs, but also for other uses.

But I think one view we both share is it would be a grave mistake to look at the world now and assume that’s how things are going to be in 2030 and 2035. The fact that you’re already seeing large numbers of people, definitely not the majority, but still large numbers of people who are non-ironically interacting with AI systems and treating them as friends, as girlfriends, as people or systems that they’re in serious relationships with—and the state of AI is nothing like how it will likely be in five years or 10 years or 15 years. So how sci-fi should we be thinking about this? What are you anticipating? What’s the world going to look like in 2030 or 2040 when it comes to this kind of technology?

Henry (15:01): It’s fascinating because I genuinely don’t know. I think it’s very hard for anyone to make super confident predictions here. At one extreme, you can imagine a world in which human identities start to become less central in our media and our discourse. I think we’re already seeing some indications of this. I think the top two songs streaming on Spotify last month were AI generated.

Dan (15:29): Wasn’t it a country song, the top country song on Spotify or something?

Henry (15:33): Yeah, I think that’s right. So as we start to see AI penetrate more and more deeply into our daily life in social media, when it comes to generated content, I can totally see a world in which my son, who’s 11 years old right now, by the time he’s 16, he’s active on Discord servers. There might be a mix of humans and bots on those Discord servers, all chatting away. And he might not even particularly care that this friend of mine is a bot, this friend of mine is human—it doesn’t really matter.

I can totally see a world in which this becomes normalized, particularly among young people. Most teenagers five years from now might have several AI friends. But that’s not the only possibility. It could be that we quickly saturate—there is a definite subset of the population who are interested in this and the ceiling on the number of people who are interested in AI companion relationships is not that high. Maybe 20, 30% of people and the other 70% just have zero interest in it. That seems like a viable possibility.

That said, I think this is a space with strong commercial incentives for creating AI companions or AI chatbot friends to cater to different niches and interests. If I had to guess where we’re headed, it’s much more widespread use of these systems, their integration more deeply into people’s social lives. I think we might well see generational divides.

I was chatting to a CME developer from one social AI product who said a couple of interesting things. They said firstly, the gender balance was surprisingly even. When it comes to early adoption, particularly of fringe technologies, men tend to overwhelmingly predominate. If you look at the data on video games or Wikipedia editors, you expect 80-20 male-female distributions. But I think in social AI, from what I understand, it’s something close to 60-40.

And there’s a big contingent of straight women who seemingly are really big users of social AI boyfriend services. This is a whole different rabbit hole we could go down. Briefly, there are some pretty clear motivations. A lot of them are coming out of toxic or abusive relationships, and an AI boyfriend gets their emotional needs met, or at least to some degree, without posing the kind of emotional or even physical safety risks they associate with other relationships.

Another point here is that this is something my wife stressed to me: the way to think about AI companions is not via the analogy of pornography, which is predominantly still consumed by men, but rather erotic fiction, which is overwhelmingly consumed by women. It’s one of the genres that has the biggest gender skew out there, something like 90-10 female to male readers. And of course, this is still predominantly a text-based medium, so maybe we shouldn’t be surprised that a lot of women are enjoying having AI boyfriends or husbands.

That’s the first thing this developer mentioned to me, that gender balance was surprisingly close. But the second thing they mentioned was that they had had massive trouble getting anyone over the age of 40 interested in these systems. They had tried to pitch them towards older users. I mentioned my dad as an example of someone—he doesn’t have an AI girlfriend that I’m aware of. His relationship with Alan is strictly platonic. But I think he’s the exception.

I can easily see a world in which it becomes totally normalized for young people. Maybe not everyone, but most young people will have various AI friends or AI romantic companions. And then people over 40 or 50 just look at them and say, “What the hell is going on? I do not understand this strange world.”

Dan (19:42): The thing that makes me think that people are in general underestimating how impactful social AI is going to be in the coming years and decades—there are three things.

I think firstly, people are really bad at predicting how much better the technology is going to get. I think we’ve seen that over the past few years. There’s this real bizarre bias where people think we can evaluate these big picture questions by taking the state of AI as it is today and projecting it into the future. Whereas I expect by 2030, you’re going to be dealing with systems that are so much more sophisticated and impressive than the ones we’ve got today, including when it comes to satisfying people’s social, emotional, and sexual preferences.

Another thing that makes me think this is going to be really massively influential is people already spend a lot of time immersing themselves in fictions designed to satisfy their social-sexual desires. You mentioned pornography as an example, but I’m always struck by sitcoms, like Friends. I enjoy Friends, like most of humanity apparently—a massively influential show—but there’s something a bit disturbing about it, in as much as it’s this product designed obviously to maximize attention and make profit, which gives you this totally mythical version of human social relationships, designed to activate the pleasure centers that we’ve got associated with friendships and romance and things like that, without any of the real painful conflict and misery and betrayal that’s actually associated with human social life. And yet people love that. They immerse themselves in it. There’s a massive audience for that kind of thing.

I really think social AI in a way is just going to be that kind of thing, but just much more advanced and much more impressive.

But there’s this other thing as well, which connects to what you’ve said with respect to the potential age difference, which is: I think one of the things that makes this a difficult topic to think about is at the moment, it seems to me at least, there’s quite a lot of stigma associated with the use of this technology. So I think if I found out that somebody was using an AI boyfriend or AI girlfriend or something, I would probably draw potentially negative inferences about them. And I think partly that’s because the kinds of people that are using these technologies now tend to be lower status in a sense because they don’t have a human girlfriend or a human boyfriend. Partly there’s just a weirdness factor.

And I think what that means is, because there’s that stigma, there’s this real reputation management thing going on where people would say, “I would never ever use social AI in a serious sense. I would never ever use romance AI, never use erotica AI and so on.” Because I think if you came out and said that you would, it would really hurt your reputation in today’s climate.

But I think actually, my suspicion at least, is that revealed preferences are going to suggest that this is going to be way more popular than people are letting on now. I don’t know, that’s how I’m viewing things in terms of the future. I’d be interested if you see things differently.

Henry (23:09): Super interesting. I think all of those are spot on. On this third point about the stigma, I think that probably already lends itself towards underestimation of the prevalence of social AI usage. There are interesting survey results where you ask young people, “Would you use social AI? Do you have an AI girlfriend? Do you have an AI boyfriend?” Responses are quite negative—”No, no, no. I think it’s weird. I never would.”

But then other responses, you ask in indirect ways and it looks like, according to figures I saw recently, 70% of young people under 18s in this US survey said that they had used social AI for meeting emotional needs. Or they were using generative AI for meeting social needs, including romance. So I think there is definitely some underreporting or underappreciation of the prevalence precisely because of this stigma.

A couple of other thoughts on this. I think there is some indication as well that by far the biggest users of this technology are young people, particularly under 18s, who obviously it’s very hard to study and probably do not—they’re not writing editorials in the New York Times about “my life with my AI girlfriend or my AI husband.” So I think there’s some underreporting there.

But as that cohort ages out as well, I expect that to reduce the stigma in the same way that we saw with online dating. There was a period when online dating was really stigmatized and yet now it’s how everyone meets pretty much.

Dan (24:49): Okay, so now let’s get into maybe a bit of the philosophy. You said that the way that you understand social AI at the moment is, at least a part of it is, you’ve got ironic anthropomorphizing and you’ve got non-ironic anthropomorphizing. Anthropomorphizing is when you are projecting human traits onto systems that don’t in fact have those traits. So if I attribute beliefs and desires to the weather or to a thermostat, that’s a case of anthropomorphizing, I take it.

I suppose it’s not immediately obvious that that’s what’s going on when it comes to advanced AI systems today in a straightforward sense, because you might think, “Okay, people are attributing things like beliefs, desires, intentions, personalities to ChatGPT.” But somebody might argue, “Look, it’s just the case that we’re dealing with sophisticated intelligence systems. So it’s nothing like these standard cases of anthropomorphizing. They actually do have the kinds of psychological states that people are attributing to them.” Are you assuming that view is wrong or are you using this term anthropomorphizing more expansively?

Henry (26:01): I’m using it more expansively. Sarah Shettleworth, who is one of my absolute favorite titans of animal cognition research, defines anthropomorphism as projecting or attributing human-like qualities onto non-human systems—in her focus on animals, but onto non-human systems more broadly—usually with the suggestion that such attributions are illegitimate or inappropriate.

But I don’t think it’s necessarily baked into the concept of anthropomorphism that it’s wrong. When I talk about anthropomorphism in this sense, I’m talking about it in the sense of basically just attributing human-like qualities to non-human things that may in fact have them.

My own view—I’ve got a new paper, a new preprint called “Three Frameworks for AI Mentality”—I argue that I think probably at least certain minimal cognitive states, things like beliefs and desires, are quite appropriately attributed to LLMs in this case. Not to be clear, all beliefs, all desires, but there are contexts in which it makes sense to say ChatGPT believes P or ChatGPT desires Q. I think particularly on things where there’s been specific reinforcement learning towards a given outcome—like ChatGPT really doesn’t want to tell me how to make meth, for example, because that’s something that’s been specifically reinforced to avoid doing.

I agree there are some cases in which at least minimal mental states are appropriately attributed to generative AI systems. That said, I think I’m also a big fan of Murray Shanahan’s idea that a lot of the correct and most informative way to interpret LLM outputs is that they’re role-playing characters. I think this is plausible when you think about how, particularly if you’ve spent any time interacting with base models, you basically give context cues to them about the kind of role you want them to occupy and then they play into that role—but they’re not robustly occupying that role. You change the context cues, they can switch into a different role. So I think a lot of LLM outputs are better understood as role play—they’re playing a character. But yeah, some mental states are completely appropriately attributed to them, I think.

Dan (28:13): Yeah, it’s interesting. And people talk about them as well, I guess it’s somewhat connected to this role-playing idea, as having personalities. When OpenAI released ChatGPT o1, there was apparently this big uproar because people liked the personality of 4.0, which was the model in widespread use preceding ChatGPT o1. And the idea is, just as human beings have different personalities, which are going to manifest themselves in terms of how you’re talking to them and we might use words like “how warm are they?”—similarly, people I think find it quite natural to attribute personalities to these chatbots as well.

I take it, I mean, I think we should say something about potential benefits and opportunities, really great things about social AI, but just to anticipate one of the worries, I think one of the worries is people are forming relationships with these systems at the moment and potentially in the future. And it’s almost like that’s psychotic in as much as there’s nothing there really on the other side.

I mean, I take your point about you might attribute minimal kinds of beliefs. You might take what Dennett would call or would have called the intentional stance towards these systems. But I think many people have the intuition, which is one of the things that’s just deeply, deeply troubling about people forming a relationship with one of these systems, is they might cash it out by saying they’re not sentient, they’re not conscious, but they might also say related to that, they don’t have any of the traits that human beings genuinely have, which are a necessary condition for forming a meaningful relationship. What’s your view about that kind of worry?

Henry (29:57): Yeah, I think it’s an interesting and important worry. Here’s how I would frame it. There’s a certain view that says, “Look, these human-AI relationships are a contradiction in terms because relationships involve two relata and they’re dynamic by nature. And these systems don’t really have mental states. They’re not really reciprocating any feelings. They’re not really making any demands in the same way that is characteristic of reciprocal relationships.”

Look, you can certainly define relationships that way, but I think there are lots of contexts where we talk about relationships with only one relatum. Think about the fact that so many people say they have a relationship with God. Now, we can debate how many relata there are in that case. But I mean, I think most of us would say at least some people who think they have this really deep and meaningful relationship with a supernatural being—there’s not really a supernatural being there. That doesn’t mean that that is not a psychologically important relationship in their lives.

Or I guess more broadly, you could think about relationships with pets. Now, of course, in some cases, we can look at clear reciprocation—people have deep and established relationships with dogs and cats and so forth. But you also have people talking about relationships with their pet stick insects or their pet fish, where it’s just much less clear that there is any kind of rich two-way connection there.

That said, I think there’s a broader worry here that I’ve called the mass delusion worry about human-AI relationships and social AI, which is that I think a lot of people are just going to look at this, particularly if it becomes a more pervasive phenomenon and say, “Has everyone gone mad?” Because there is no other person there, there is no other. These people are investing huge amounts of time, emotional energy, potentially money into these pseudo relationships where there’s no one on the other side.

I think that’s a case where questions about AI mentality maybe become more important. You might say, whether or not the mass delusion worry is on the right line is going to depend on the degree to which these things do have robust psychological profiles, really instantiate kinds of mental states.

I think that’s another related worry, which is—again, we can talk more about the specific psychosocial risks—but another worry I’ve heard is that even if you could prove to me tomorrow that human-AI relationships are generally beneficial for users, they make them more connected to people around them, they do all these other good things, they just can’t instantiate by their very nature the same kinds of goods that human-to-human relationships instantiate. This is very much a philosophical rather than psychological point. It’s like they’re just the wrong kind of relationships to have the welfare goods that we associate with human-human relationships.

Dan (32:36): Yeah. Just on the God analogy, I mean, as an atheist, I really do think people are taking themselves to have a relationship to something that doesn’t exist. And from my perspective, that’s a deeply objectionable aspect of the practice that they’re engaged in.

The analogy with pets is very interesting. One of my most unfashionable opinions—maybe we’ll have to cut this bit out because it’s going to be reputationally devastating—I do think it’s kind of, I find it a little bit humiliating how some people relate to their pets, the degree to which they anthropomorphize them. That’s not to say that you can’t have deep relationships with pets. Clearly you can. And I love dogs, for example, but I think there are cases where people treat their dog or their cat as if it’s a person. And it’s not that I think it’s psychotic, but I think there’s something objectionable about it. I think they’re making a deep mistake. And even if they’re getting psychological benefits from that, there’s something deeply, almost existentially troubling about that kind of relationship.

And I can see that mapping onto the AI case, except I take it in the AI case, and this connects to one of our previous episodes, people’s intuition is, well, with a dog, maybe it’s not cognitively sophisticated, but just about everyone is going to assume these days at least that dogs are conscious, sentient is the term that is often used in popular discourse. And that does change things in a sense. Dogs really care about things. There’s something it’s like to be a dog. Whereas with AI systems, I think many people have the intuition that they might be informationally and computationally sophisticated, but there are no lights on inside. There’s no consciousness there. And that changes things again.

Anyway, that was just my sort of immediate reaction to these analogies.

Henry (34:33): I think I’m sympathetic. I think certainly when you hear people talk non-ironically about fur babies and so forth, it does seem like there is some degree of maybe inappropriate allocation of emotional and relational resources into certain kinds of relationships. And I say that as someone who adores animals, has had dogs most of my life.

Maybe another couple of examples of non-standard relationships. I think a lot of people would say they have ongoing relationships with deceased relatives. Particularly if you’re coming at it from a spiritual point of view where you believe that your deceased relatives are looking over you, seeing what you’re doing, or whether you understand it in some kind of more—animist isn’t quite the right word—but you still think about “my ancestors are smiling at me, Imperial. Can you say the same for yours?”—the famous line from Skyrim. A lot of people have this sense of “Yeah, my ancestors are looking over my shoulder. I’ve got to live up to their expectations for me.”

So I think there are lots of interesting cases where we do have these relationships that don’t meet the canonical definition of this highly dynamic, reciprocal, ongoing kind of relationship. Also, I’ve got friends that I would consider myself to have a valuable relationship with that I haven’t spoken to in three years in some cases.

So I guess all of which is to say, the category of things that we call relationships is weirder and bigger than might meet the eye if the only notion of relationship you’re working with is like, “Yes, my wife or my friend who I go to the pub with three times a week.”

Dan (36:13): Yeah, that’s interesting. Okay, I mean, I really want to spend probably the bulk of the remainder of this focusing on potential threats and dangers here. But I think it is worth stressing that this as a technology, social AI, does have enormous benefits and opportunities associated with it. I take it even if you think that there’s something troubling or objectionable about people having these relationships, some people are in such dire circumstances of loneliness, of estrangement from other people for whatever reason, potentially because they’re in old age where I think issues of loneliness are really prevalent. And clearly, under those conditions, I think social AI can be enormously beneficial in as much as it just makes people feel much better than they otherwise would.

It also seems to be the case that, at least in some cases, people are using social AI to hone skills and acquire confidence that they then use when it comes to interacting with people in the real world. And I can completely imagine that is a current benefit of this technology and it’s likely to be a benefit which in some ways will get amplified as we go forward in the coming years and decades. Are there any others that are missing there in terms of real positive use cases of this technology?

Henry (37:39): Yeah, I think you’ve nailed some of them. One thing I would just really stress here, there’s a move that drives me nuts, where people basically say, “Well look, even if you’re using chatbots to alleviate your loneliness, it’s not fixing the root cause.” It’s like, okay, well, okay, yeah, go away and fix the root cause—solve human loneliness. Please come back and tell me when you’ve fixed it. So classic case of letting the perfect be the enemy of the good. Loneliness is a pervasive problem and arguably one that’s getting worse, although the social science of this is messier and more complicated than you might think. But we don’t have a magic wand that can cure loneliness.

And the question then becomes, I think, is this actually just in the short term making people’s lives better or worse on average? And I think the data here, this was a big surprise to me, the limited data that we have suggests that most users of social AI, at least that show up in studies, report significant benefits from this.

I can just quote from a couple of studies. This was a study by Rose Gingrich and Michael Graziano from 2023 looking at Replika users. They found that users generally reported having a positive experience with Replika and judged it to have a beneficial impact on their social lives and self-esteem. Almost all companion bot users spoke about the relationship with the chatbots having a positive impact on them.

Another interview-based study in 2021 found that most participants said that they found Replika impacted their well-being in a positive way. Other sentiment and text mining analysis studies have found really similar patterns.

So right, I think the data right now, which is very limited, got to stress, it’s very imperfect, supports the idea that most users report benefits from the system, net benefits.

Now, just to flag some of the problems: these are typically self-selected subjects, they’re cross-sectional studies so we’re not looking at the long-term impacts of this technology on their lives, and we’re relying on self-report measures. If you asked me “Does video gaming have a positive impact on my life?”, I’m absolutely going to say yes. You ask my wife—I think she’ll probably say yes as well—but the point is that people tend to justify their own life decisions so they’re unlikely to say “Yeah, this thing that I dedicate a dozen-plus hours a week to—yeah, it’s really bad for me. I shouldn’t do it.” That would require a level of brutal self-honesty that maybe people don’t have.

So I just say that as a caveat, but I equally think we can’t ignore the data points that suggest that most social AI users do experience net benefits currently.

That said, I also think we should be very aware of a whole host of potential downsides. You mentioned this idea that using these tools could help people cultivate social skills or regain social confidence. But equally, of course, you have the flip side worry about that—this idea of de-skilling that comes up in a lot of technology.

De-skilling, in case anyone’s not familiar with this, probably one of the most widely studied domains of de-skilling is in aviation, where we’ve had serious airplane crashes that have been linked to pilots’ over-reliance on automated instruments rather than being able to fly just manually. And this has led to them not acquiring relevant kinds of piloting skills such that when the instruments go wrong, they don’t know what to do.

And you might think something similar could happen socially. If you have this generation of, particularly, I think we’re looking at it through the lens of young people—if you have a generation of young people who are primarily interacting, a huge proportion of their interactions happen with bots that are maybe more sycophantic than humans, they’re always available, they never interrupt you and say, “Actually, can we talk about me now for a bit?” Or “Look, this is very interesting to you, but come on, give me a turn”—they don’t accurately recapitulate the dynamics of human-to-human interaction. You might worry that that would lead to people failing to acquire relevant social skills.

Another big worry that I think is a nice intersection of our interests is the potential for these things to be used for manipulation and persuasion. Particularly again, I think young people is quite salient. If I’m a young person with a chatbot, I’ll give you a really simple example. If I’m going to ask my AI boyfriend or girlfriend, “I’m getting my first mobile phone next year. Should I get an Android or an Apple phone?”—well, I mean, that’s a lot of power you’re giving to the bot.

Now, of course, this is a problem with LLMs more broadly. You might think that that’s a lot of leverage that’s in the hands of tech companies. But I think, and I think we’re probably broadly on the same page here, to the extent that you think that in domains like moral and political views in particular, social factors are incredibly powerful, and then you think the AI love of my life, my AI girlfriend is telling me, “You should vote for Trump” or “You shouldn’t vote for Biden” or “You shouldn’t listen to that kind of music, it’s uncool” or “You should be interested in this kind of music”—all of this stuff could exert a much bigger influence than just asking ChatGPT, precisely because you’ve got those social dynamics in place.

Dan (42:46): Yeah, I think that sounds exactly right. I mean, I think probably the complicating factor there is if you imagine a corporation that wants to maximize market share, maximize profit by producing technologies, the function of which is to satisfy people’s social needs specifically, and then it comes to light that there’s also this manipulative, propagandistic agenda at the same time—either because in the midst of your loving relationship with your AI, it starts saying, “Well, you should vote for Reform or Labour at the next election,” or because there’s some news story that comes to light which shows that there’s nefarious funding or influence behind the scenes—I can imagine that would be really catastrophic for the business model of the relevant corporation. And that kind of thing, I think, is just really difficult to anticipate.

Henry (43:31): Yeah, potentially. So I think there are some incentives that companies have not to be too crude about leveraging human-AI relationships for cheap political or commercial gain. But I can also imagine in some contexts—I think not to pick on China here specifically—but I mean, social AI is huge in China. We haven’t talked about that, but there’s a service called Xiaoice that has, according to some estimates, several hundred million users. They themselves claim they have 500 million users. It’s a bit more complicated than that because Xiaoice is a whole suite of different services. We don’t know what proportion of those have active ongoing relationships with the chatbot girlfriend/boyfriend component.

But part of Chinese AI regulation says that the outputs of generative AI systems have to align with the values of communist China, the values of socialism. So you can imagine a generation of young people who have these deep relationships with AI systems. Those AI systems, for legal compliance reasons, have to basically align with the broad political values of the incumbent regime. And they will dissuade or deter people from maybe exploring alternative political views as a result. So maybe that’s the more subtle kind of influence, rather than just like, “You should vote for Trump because he gave our social AI company X million dollars.”

Dan (44:57): Yeah. And I think that’s just part of a broader fact, which is that propaganda and manipulation and so on, they do in fact just work very differently and sometimes much more effectively within authoritarian regimes than they do in more democratic, liberal ones.

On this issue then of potential costs that the use of these technologies might generate when it comes to then interacting with people. So I think you mentioned the fact that these systems, they’re almost going to be more sycophantic by design, at least if you assume that at least up to a point, that’s the kind of AI agent that people are going to enjoy interacting with. And they don’t have all of the sources of conflict and frustration and misery that go along with human relationships.

So one issue is to the extent that people start using social AI much more, they’re going to lose precisely that skill set, which is adapted to dealing with other human beings who haven’t been designed to cater to your specific social preferences.

But I take it there’s also then an issue of motivation, where it’s sort of, why would I go out there into the world and spend time with human beings with their own agenda and their own interests who are frustrating and annoying and often insulting, etc., etc., when I could immerse myself in this world in which it’s pure gratification of my social, even my romantic, sexual desires?

I think that then connects to something else. When I think about this area of social AI, the thought experiment that seems most salient to me is the experience machine—Nozick’s idea—would you plug yourself into some machine where you’re getting all of the wonderful pleasure that goes along with certain kinds of desirable experiences, but none of it’s real? You’ve just been fed the relevant kinds of neural signals to simulate those kinds of experiences.

I think many people think, no, because there’s much more to a meaningful life than merely the hedonic, affective associations or things that are associated with the satisfaction of our desires. We actually want the reality that goes along with satisfying our desires. And I think similarly, when it comes to social AI, the intuition is there’s something similar going on there where it’s okay, you might be getting off socially, romantically, sexually, and so on, but it’s fake. It’s not reality. And so even if you’re the happiest person in the world from a certain definition of what happiness amounts to, which is purely hedonic, there’s nevertheless something deeply troubling about that kind of existence which we should steer ourselves away from.

What are your thoughts about that? Do you think that analogy with the experience machine thought experiment makes sense? And do you also buy the intuition that we shouldn’t plug ourselves into an experience machine, so we also shouldn’t plug ourselves into very pleasurable forms of social AI?

Henry (48:06): Super interesting. I’m going to tease apart two different threads here. The first idea is this idea that these are just going to be easier alternatives. And I think the useful lens for thinking about that is the idea of superstimuli. This is a term that gets thrown around in a lot of different domains. We hear it in relation to food—that modern junk food is like a culinary superstimulus that basically just gives you far more rewarding signals associated with fat and sugar than anything you would find in our evolutionary environment. And this has sometimes been suggested as the best explanation for the obesity crisis—the fact that basically modern food is just so delicious, it just maxes out our reward centers so effectively that it’s just really, really hard to go back to eating whole grains and leafy vegetables cooked simply.

We see the same debate around pornography, the idea that pornography is kind of a sexual superstimulus. I’ve also heard the superstimuli used to refer to things like social media or short-form video. Try reading a Dickens novel if your brain has been fried by a decade of six-second YouTube Shorts.

Now we don’t need to relitigate that, but I think all of those debates, this idea of superstimuli, the idea of something that is just far more rewarding than the kind of natural, maybe more wholesome version of it—I think that’s a really powerful lens for thinking about social AI and raises some significant concerns.

But I think that’s separate from the second point you raised about the idea that it’s fake, that the actual thing that we value is lacking in these kind of contexts.

For what it’s worth, I am far more conflicted on the experience machine, I think, than you. There is a sense in which I think I would be very tempted to take the experience machine, although maybe that has to do with the fact that I’m pretty sure we’re in a computer simulation right now. I’m hardcore into simulationist territory.

I also think there are maybe some reasons we should expect our judgments about the experience machine to be maybe skewed. We have this idea of real goods versus ersatz goods, where real goods are going to be more enduring, more reliable. So that might create within us a preference for real goods over fake goods. But of course, in the experience machine, you’re guaranteed—these experiences will keep on going. You’re going to have this dream life in the matrix or whatever, that’s not going to be yanked away. So I think there are intuitions that possibly make us more averse to experience machine type lives than we possibly should be.

Dan (50:54): Well, there’s also, just to really quickly interrupt on that point, Henry, sorry to interrupt, but there’s also, I think, as I mentioned earlier on in connection with social AI, this reputational thing going on where I think there’s a tendency to judge people harshly if they choose the experience machine, potentially because we think somebody who’s going to prioritize positive hedonic experiences wouldn’t make for a good cooperation partner or something. I think we’re constantly evaluating, would this be a good leader, a good friend, a good romantic partner, a good member of my group? And if someone seems to suggest that they would prioritize mere hedonic manipulation or however exactly to understand the experience machine, we judge them harshly. And I think anticipating this, then people are inclined to say, “No, I wouldn’t choose the experience machine. I would choose the real thing.”

I just also want to really quickly talk—you went over this quickly, but I think it’s very interesting. You said you think we probably are living in a simulation. So this is the classic Bostrom-style argument that says, well, we’re likely to be able to have the technology to create simulated worlds. If that’s true, then there are going to be many, many more simulated realities than base reality. So just statistically or probabilistically speaking, we should assume ourselves to be in a simulated reality.

But as I understand that, that’s in and of itself, it’s not obvious to me why that would influence your response to the Nozick experience machine scenario. Because even if you think it’s true that we are living in a simulation in some sense, I take it what’s distinctive about the experience machine thought experiment is it’s not just that you would be living in a simulation. It’s well, for one thing, it’s a simulation within a simulation, but it’s also a simulation which has been tailor-made in a way to satisfy your desires and that feels a little bit different from what I’m assuming you take to be the case when it comes to us living in a simulation. Did any of that make sense?

Henry (52:55): Yeah, it makes perfect sense. Okay, here we can get really spicy because I think I take Bostrom’s classic simulation arguments quite seriously, but at the risk of having viewers think I’m completely bonkers, I’m also genuinely—I think there’s a chance I’m already in an experience machine. So this is just speaking for me.

Without wanting to get too sidetracked, I think so many features of my life—it’s hard to put this without sounding egotistical, but I just feel like my life has just been absurdly fortunate. I’ve lived in a really interesting time in human history. My life has been blissfully devoid of serious unpleasantness. Not to say that it’s been perfect, but most of the challenges I’ve encountered in life have been interesting, relatively tractable things. Look, here I am. I’m getting to live the life of a Cambridge philosopher at the very cusp of human history where we’re about to explore AGI. It seems like this is the kind of life, genuinely, that I might pick.

Whereas I feel like by rights I should have been a Han peasant woman in third century AD China. So I’m slightly joking, but not entirely. I do think there’s a serious chance that at some level I am already—my life consists of some kind of wish fulfillment simulation. So I don’t know, maybe that gives more context.

Dan (54:17): Interesting, okay, that’s a spicy take. We should return to that for another episode because that’s fascinating. And actually, it’s also very philosophically interesting. I think you’re right, when you start thinking about things probabilistically in that way, yeah, the fact that you’re having a great life might provide evidence for this kind of experience machine scenario. But I feel like we’re getting derailed from the main point of the conversation now, which I think is probably my fault for double clicking on that point.

Henry (54:45): So there’s this question about whether the fact that there’s maybe no one conscious on the other end of the conversation—to what extent does that mean that it fails to instantiate the relevant goods that people care about? To what extent does that make it fake?

I think one small thing, I think there’s a perfectly viable position that says, look, a perfectly viable philosophical position that says, look, if there’s no consciousness there, then no matter how much fun these relationships are, they’re not really valuable. I mean, I can sort of see some arguments for that.

But maybe I’m also influenced here by the fact that really since my early adolescence, so many of the relationships I’ve had—I’m talking relationships in the broad sense, friendships and so on—have been with people whose faces I never saw. I spent from my early years a lot of time in pre-Discord chat rooms on services like ICQ back in the day, on online video games, massively multiplayer worlds, before the age of streaming video, where I would form these really valuable relationships with people I only interacted with via text. And in some cases, particularly in video games, we didn’t really discuss our personal lives at all. We were just interacting in these virtual worlds.

Now you could say, yeah, but there really were people, conscious people on the other end there. And sure, yeah, but I’m not sure how much of a meaningful psychological difference that makes for me in terms of my experience of those things. It seems to me that so many of the goods that we get from relationships don’t consist in this deep meaningful connection with a conscious other but consist in things like joint action—in the case of a video game might be going on raids together, having some fun banter together, discussing politics.

I spend a huge amount of time—or not so much these days, back in the day I spent a huge amount of time arguing on Reddit. Would it have made a difference to me if I knew that person I had a really good long political debate with on Reddit, that they were a bot? Well I think it probably would, but I’m not sure whether it should, if that makes sense. It was a valuable discussion for me and maybe it’s just my own prejudice that gets in the way then.

Dan (57:04): Interesting, yeah, I definitely don’t have the same intuition. I think if I were to have what I thought of as meaningful relationships and then discover that actually I wasn’t dealing with a person as I understand it—

Henry (01:03:47): Yes, okay, if you found out that you were talking to a chatbot online and you thought they were a person, that would be dismaying. You would feel bummed out to some extent.

It’s not clear to me that that is primarily to do with lack of consciousness though. In some ways, I think it’s more to do with a loose set of considerations around agency and identity. Talking to a chatbot feels like something like a social cul-de-sac currently. There’s no one going to go away with changed views that they’ll carry forward into discussion with other people as a result of the conversation that we’ve had. It can feel sort of masturbatory in that sense.

But I think if you can think about AI systems, if we imagine AI systems as something like more robust social agents—so you might chat to a given social AI one day and that social AI will be able to carry forward any insights it gleans from that conversation into interaction with other people. I don’t know, as you beef up the social identity of these things a bit more so that it’s not just these masturbatory cul-de-sacs, then my intuitions start to weaken a bit. Maybe they can be valuable. If I’m talking to a chatbot that speaks to other people and can carry forward those insights, maybe there is some value.

Dan (01:05:06): Yeah, that’s so interesting. I feel like there’s a million different things we could be talking about here. And we should say we’re going to have other episodes where we return to social AI, where we bring on guests and so on.

Maybe two things to end on. One thing we’ve already touched on, and I think it connects to what we’ve just been saying, but takes it even further. One commercial use of social AI, at least as I understand it, pretty fringe use, but not a completely non-existent one, is I think you earlier on called them grief bots? Basically people using AI technology to produce a system that—I don’t exactly know how to describe it, but that exemplifies the traits that they associate with somebody, a loved one, a family member, a spouse, a friend who has passed away.

I mean, that it’s almost like that’s been cooked up purposefully for moral philosophers because it introduces so much, not just weirdness, but moral complexity. I mean, I take it there’s the general baseline issue, which is you’re forming a relationship of some kind, you’re interacting with an AI system, and that’s weird. And then it’s a weirdness which is massively amplified by the fact that you’re interacting with an AI system, but the relationship in some sense is grounded in the perception that you’re somehow interacting with somebody who’s now deceased.

I don’t even know to be honest how to describe it, but I understand some people are doing this. So what’s your take about what’s going on?

Henry (01:06:51): Yeah, so I expect griefbots to be one of the big applications of social AI. As you mentioned, it’s relatively niche at the moment. And I think a lot of companies are very scared to go anywhere near this, but I can see that changing quite rapidly.

Just for context here, it’s worth noting that chatbots fine-tuned on real-world individuals—there’s a lot of them that suggest they can be really accurate in terms of capturing the kinds of things that people would say, their modes of conversation and so on.

So one famous experiment was the DigiDan study. This was using GPT-3, so really primitive language model by modern standards. But a group of people including David Chalmers, Anna Strasser, Matt Crosby and others basically fine-tuned GPT-3 on the works of Daniel Dennett—I’m sure most of your audience know Daniel Dennett, one of the greatest philosophers, sadly died a couple of years ago. This was shortly before his death though that he did this.

And then they got Dan’s friends and colleagues to pose questions to both Dan himself and the DigiDan bot. And they generated four responses from the chatbot and they had Dan’s response there as well. And users, Dan’s friends and colleagues, were pretty much close to baseline, close to bare chance, telling which responses were from Dan versus the chatbot.

So this is just to emphasize that appropriately fine-tuned chatbots could do a really good job of simulating the kind of things that a person would say in response to a given query.

So let’s just imagine that you do have this category of griefbots that can provide an accurate simulacrum of a deceased person. Well, I mean, at the risk of spicy take, I can see lots of really positive use cases for this.

I’ve actually even said to Anna Strasser, one of the people who did the DigiDan study, that if I get hit by a car tomorrow, I’ve discussed this with my wife as well, then she should absolutely have my permission to fine-tune a bot on me. And my wife will give her a perhaps lightly edited or curated set of my correspondence, my social media presence and so on, so that then my kids can talk to this simulacrum of me if they choose to.

I can imagine there being real value, know, if my son or daughter is like 17 and considering, should I go to law school or medical school? I wonder what my dad would have thought of this. That seems like a really potentially positive use case.

So I think griefbots are a really interesting area. And there’s interestingly some studies looking at people’s use of griefbots for therapeutic purposes—if there are conversations you always wanted to have, but didn’t get the opportunity to have perhaps because a spouse or a parent died suddenly, that this could, one phrase that’s used, offer a soft landing for the grief experience. So there are loads of really interesting positive use cases there.

But equally, as you say, it’s an absolute minefield. And I think to a lot of people the whole idea of griefbots just feels like something from a Black Mirror—well, literally there was a Black Mirror episode about this. And I think that it raises some fascinating questions about how that changes the nature of the grieving process, how it changes our very concepts of mortality.

If someone’s physical body can die, but there’s this sort of echo, digital echo ghost of them that is still around, how does that reshape our views about these things?

There’s also some interesting parallels. I had a student who wrote a great dissertation or a great essay about integrating this with the idea of communing with ancestors, which is obviously a really common feature in many different societies, where you might ask your ancestors—we touched on this earlier on—you might ask your ancestors for guidance on difficult questions. Could griefbots be a way of making that into a more concrete experience?

There’s a second angle here, which is even spicier and again, will make viewers think I’m even more of a weirdo, which is: could this actually offer some kind of form of immortality or some kind of life after death? Or continued existence after death, I should say.

Now, we could do a whole episode on this as well, but for what it’s worth, as someone who is very deflationary about personal identity, I’m big on the work of philosophers like Derek Parfit who say that in some sense the self is an illusion or the persistent self is a constructed self, there’s no deep matter of metaphysical fact about whether I survive or not. I could see a good case being made that in some sense, via an appropriately fine-tuned chatbot, there will be a form of persistence of me through that chatbot that might be relevant to mortality considerations.

Dan (01:11:36): Yeah, in some sense. It seems like there’s an issue here, which is using a chatbot to acquire knowledge about what a given person might have thought about a topic. And I take it there are going to be all sorts of questions that arise there. Like to what extent is it going to be reliable as a way of gaining insight into what that person would have thought about a topic?

Then there’s a question of using these systems not just to get that kind of knowledge, but to actually have a kind of relationship with the person. And then there’s something over and above that, which I take it you’re referring to then, which is the idea that in some sense, such a chatbot would carry on the identity of the relevant person. And I take it that relationship thing and that identity thing are connected.

I mean, certainly I think that’s where the real issues and lots of people’s queasiness arises, right? That’s when it seems like a bit of a leap, at least if we’re talking about chatbots as we understand them today. I can imagine AI systems of the future that aren’t merely getting really good at the statistical pattern recognition and prediction when it comes to bodies of text, but that are doing something more substantial when it comes to replicating the characteristics and traits of the relevant person.

But do you really think if you had a chatbot that had been trained on the text that you had produced that it would be in any sense a continuation of you?

Henry (01:13:08): Yeah, I mean, potentially. We could do a whole episode on personal identity here. But broadly speaking, being a bit crude here, the Parfittian perspective, Derek Parfit’s view, is that there’s a certain kind of relation you bear to your future self, a certain kind of psychological relation, that can come in varying degrees. And to the extent that we prioritize anything when we’re about survival, this relation R is the kind of relation that matters. And it can obtain to varying degrees.

Just as I could suffer a traumatic head injury and my behavior would change in some ways, but not others—that would be a sort of continuation of me in some ways, but not others. I think you could say the same for an appropriately fine-tuned chatbot.

Now, as you sort of implied, there will be things that it misses out there. We don’t talk about everything that is relevant to us. There’s more to our identity than just what we say. But again, that seems like a technological problem that, as we move to increasingly multimodal chatbots that can learn not just from what we say online, but how we live in the world, I think you can instantiate this relation, this relevant kind of continuation relation to increasingly strong degrees.

But I guess that’s the central point I’d say here: that survival in this kind of Parfittian view is a matter of degree and a matter of similarity and I don’t see why even if it’s imperfect—to the extent that a chatbot can capture really key features of my modes of interaction—that that’s a kind of survival. That’s a kind of continuation.

Dan (01:14:39): Interesting. Maybe we could end on this point. So I’ve been thinking quite a bit recently about how advances in artificial intelligence might gradually eat away at human interdependence. When people are thinking about the dangers posed by AI, there are the classic loss of control, catastrophic misalignment dangers that we’ve talked about previously. There are also dangers to do with elites, political factions, authoritarian regimes, the military using advanced AI to further objectives in ways that are bad for humanity.

But I think there’s also a category of dangers associated with AI systems doing what we want them to do, satisfying our desires, but in ways that have really knock-on bad consequences. And it’s easy for me to see how social AI might be a little bit like that, in as much as so much of our understanding of the human condition, so much of the societies that we inhabit is bound up with interdependence. We depend upon other people. We depend upon other people for friendship, for labor, for sex, for romance, for art, for creativity and so on.

And it seems like a very plausible path when it comes to advances in AI is these AI systems are just going to get better and better at doing everything that human beings do and in fact are going to get better than human beings at doing all of those things, including when it comes to satisfying the social needs in the way that we’ve talked about in this conversation.

And to the extent that that’s true and we become more and more reliant on these AI systems and less and less reliant on other people, that kind of human interdependence fades away. I think there’s something—I mean, there’s something disturbing about that just from the perspective of thinking about what it means to be human. But maybe that’s not really a serious philosophical worry. That’s just an emotion.

But I also do think, the way I think about it is a lot of the human alignment problem—like how do we align our interests with one another and build complex societies—is precisely this interdependence. Because we depend on other people, we have to care about them and we have to care what they think about us and so on.

And it seems to me one of the diffuse risks and long-term risks associated with social AI is precisely that, that as this technology gets better and better, it’s just going to erode that interdependence, which is really central to the human condition.

I realize that’s a massive thing to throw at you for the final question, but what are your thoughts about that? And then we can wrap things up.

Henry (01:17:15): Super interesting. Yeah, so I think you did this yourself, but I’ll separate out two different concerns here. One is the more philosophical question about, even if this works perfectly, even if we’re all very happy with this future society, has something of value been lost? I think it is a valuable question to ask, but it’s also one that’s hard to answer in a neutral sense. It really comes down to what is your conception of eudaimonia, human flourishing, in a deep philosophical sense.

And then there’s—but I felt like that wasn’t the core of your question. You were asking something more about negative externalities, perhaps more about negative knock-on effects. And I think that is absolutely something I’m also worried about basically just because of social media.

Now I realize this is a debate where you have your own very well-developed positions, but I’ll just offer a quick parallel of two technologies. One is violent video games or video games in general. I think we’ve probably discussed this before, but back in the 90s, there was massive moral panic around negative knock-on effects. The idea that kids growing up in the 90s playing Doom or GTA would turn into moral monstrosities as a result of being exposed to relatively accurate simulated violence.

I don’t think that was a stupid thing to worry about. It just turned out to not be a major concern. It turns out that’s not how the brain works, that we didn’t see massive negative externalities associated with exposure to violent video games.

But by contrast, social media, opposite story. I think there was relatively little panic early on in the days of social media, in the early days of MySpace and Facebook. In fact, I think most of the commentary about social effects of these things was quite positive—the idea of bringing people together. There was an interesting debate that’s quickly been sort of consigned to the dustbin, quickly been memory-holed. People, I remember in 2010, people were talking about how social media meant the collapse of epistemic closure, how these epistemic islands would be all beautifully linked up through conversations on social media. And we’d be able to talk to people with different political views from us.

And that’s basically not happened, that social media exacerbated some of our worst social tendencies, possibly contributed to echo chambers and so forth. I’m aware that you have a slightly more optimistic view here, but I’m just offering that as a parallel for a case of social technology that I think at least in many people’s view has had significant, largely unforeseen negative consequences.

And I think that’s absolutely a legitimate source of worry about social AI. I’m not sure I’d necessarily frame it in terms of dependency or interdependence. I mean, I think different people make different choices about how much they want to depend on others. I don’t think it’s obvious that someone living alone on a ranch in rural Texas or whatever, they grow their own food, whatever, they’re relatively autonomous and independent—it’s not clear that they’re not living a great life. It seems that you can have valuable lives with varying degrees of social interaction and dependency on others.

But at the same time, I do think there are possible dangers, very hard to predict, associated with potentially people getting into islands of social activity where it’s just them and their coterie of AI friends and they don’t see the need to interact with others. The kind of subtle influences that could have on things like democracy, on society—there’s absolutely scope for concern.

Dan (01:20:52): Yeah, okay. We’ve opened several cans of worms to conclude the conversation. I’m aware, Henry, that you’ve got a place that you need to be. So that was so much fun. So many issues and questions, which I feel like we didn’t really even scratch the surface of, but we’ll be back in a couple of weeks to talk about more of these issues. And then over the future, we’re going to bring on various kinds of guests and experts in social AI and connected issues. Was there anything final that you wanted to add before we wrap up, Henry?

Henry (01:21:24): I guess just a couple of quick reflections. Firstly, I think for any young philosophers or young social scientists listening, I think this is just such a rich and underexplored area right now. There are so many interesting issues ranging from griefbots to digital duplicates—models fine-tuned on real-world individuals who are still alive—to issues around de-skilling, dependency, mental health, atomization, loneliness, intellectual property, influence, motivation, persuasion. There’s enough for several dozen, hundreds maybe of PhD dissertations on this topic. So I think it’s just a really interesting and valuable area to work on.

And that’s not even getting into the meatier philosophical issues we sort of just touched on briefly around personal identity, what it means to be human, flourishing, the good life. So I just think this is a really valuable area.

Also worth quickly promoting that I am a unit editor for an Oxford University Press journal series called “AI and Relationships.” It’s called the Intersections Journal Series and I run a project in there. So if any young philosophers or academics have papers on this, feel free to give me a ping on Twitter or to my email if you’ve got anything you want to publish on this topic because I think, yeah, it’s an area where I’m really keen to start seeing more good research.

Dan (01:22:41): Fantastic, yeah. It’s a golden age for philosophy, which is why it’s a little bit strange when you look at so many of the things that philosophers are actually working on. But anyway, that was great. See everyone next time.