In this conversation, we talk about this controversy and the claims that triggered it, including:

• What agentic AI tools like Claude Code and Codex can already do for research, from coding and data analysis to literature reviews, translation, and brainstorming, and why only around 20% of quantitative social scientists currently use them.

• What best predicts whether researchers adopt or reject AI: ignorance, openness to experience, methodological background, or the awkward role of self-interest.

• How much published academic research is genuinely mediocre, and whether the cause is laziness, lack of skill, or a broken incentive structure, with a detour through the replication crisis and some high-profile fraud cases.

• Whether AI will raise the quality of research or simply flood the literature with more slop, and what journal editors could do about it.

• Whether AI can be genuinely creative or only recombine what already exists, by way of Margaret Boden’s three kinds of creativity, Thomas Kuhn on paradigm shifts, and AlphaGo’s “Move 37”.

• The fight over AI writing and detection tools like Pangram, and why current disclosure norms end up punishing the honest.

• The angry response to Alex’s series, and what is really driving reflexive opposition to AI among academics.

Links and further reading

1. Alexander Kustov — Alex’s homepage, with an overview of his research on immigration, public opinion, and effective governance.

2. Popular by Design — Alex’s Substack on public opinion, persuasion, and the politics of getting good ideas adopted.

3. Academics Need to Wake Up on AI — followed by a Part II and Part III

4. Pangram — the AI-detection tool discussed at length, which labels text as human, AI-assisted, or AI.

5. AlphaGo versus Lee Sedol — the 2016 match, including the famous “Move 37” that Henry raises as a candidate for genuinely transformative machine creativity.

6. Margaret Boden — the cognitive scientist whose distinction between combinational, exploratory, and transformative creativity frames part of the discussion.

7. The Structure of Scientific Revolutions — Thomas Kuhn’s account of normal science and paradigm shifts, referenced in the exchange about AI and discovery.

8. “AI Is a Better Researcher Than You” — The Chronicle of Higher Education‘s account of the controversy around Alex’s series.

Transcript

• Please note that this transcript is lightly AI-edited and may contain minor mistakes.

Dan Williams: Welcome back. I’m Dan Williams, and I’m back with my co-host, Henry Shevlin. Today we are honoured to be joined by Bluesky’s favourite academic, Alexander Kustov. Alex is a political scientist at the University of Notre Dame and the author of one of my favourite Substacks, Popular by Design. His primary research is on immigration and public opinion, but that’s not really what we’re going to be talking about today. We’re going to be talking about a fascinating and hugely viral series he published at his Substack titled “Academics Need to Wake Up on AI,” about what AI can already do when it comes to research, and what that means for the academics who are not paying attention, which is many of them. It was very widely read, and it generated, let’s say, a somewhat polarised response. So Alex, to kick us off: what’s the central thesis of this series, and what motivated you to write it?

Alexander Kustov: Thanks, Dan, for having me. I’m a huge fan of the Substack and the whole podcast series with you and Henry. So, like some of us, I’ve been using some of these AI tools. I’ve been reading some of the other folks like yourself, and it really transformed everything I do in my life. And I should say I was also on sabbatical, so I had a little bit more time than some of my colleagues to try some of these tools. I just hadn’t really seen any of my colleagues talk about it. And when they did talk about it, they usually tried not to be vocal about it. I just didn’t think it was a good equilibrium, where basically people were using these tools to be ten times more productive and not talk about it. It really heightened this sense of inequality for me, which I do care about. You’d have a situation where someone would publish ten papers in a year and someone else would publish one, and the only difference is that the person publishing more is the one using Codex or whatever. I just wanted to write about it. And I saw that the prevailing academic discourse on the issue, especially on platforms like Bluesky, was very counterproductive.

I didn’t really say much, to be honest. I didn’t think it would be that controversial. But the biggest thesis that really rubbed people the wrong way was that right now a lot of these tools are better at a lot of the tasks that we do as professors. I’ve refined this idea a little bit, going back and forth with some of my critics, but I feel comfortable right now saying that if you look at it globally, and think about what professors do around the world, in social science and adjacent fields especially, AI agentic tools can do most of the tasks they do in terms of literature review, data analysis, and even coming up with some research questions, better than those professors on average. I think that’s a pretty uncontroversial statement at this point, but obviously a lot of people were very, very upset about it.

Dan Williams: Empirically speaking, it is a controversial statement, in the sense that it provokes controversy when you say it. In a minute we can get to the question of what AI can actually do in the context of research. But for what it’s worth, I completely agree with you that on many tasks AI is clearly better than what human beings can do. Is your sense that lots of people just weren’t aware of that fact, that they literally didn’t have exposure to these tools? Or was your sense that the reason people weren’t really talking about it is because of all the controversy surrounding the use of these tools, not just mere ignorance?

Alexander Kustov: I think it’s both, for sure. There was recent research done by Anthropic. They tried to do, not a representative survey, because obviously the population is very hard to define here, but they surveyed something like 1,200 quantitative social scientists, and the estimate right now is that about 20% of folks use agentic tools. That doesn’t seem like much at all, and if anything it’s probably an overestimate, because they’re more likely to tap into well-resourced universities. So I do think it’s both: the little uptake we have, and the fact that people who do use these tools don’t want to talk about it.

There are two things here. First, you want to maintain your comparative advantage. This moment right now is exactly the moment where, if you’re one of the few people using these tools, you can write a bunch of papers and get tenure while the tenure system is still in existence. And the other thing is that if people are very upset about anything AI-related, you don’t want to talk about it and be shamed by your colleagues. Just to give you one funny anecdote: at the height of the vitriol I experienced, where hundreds of people literally were quoting me and trying to tag my employer to get me fired, the exact same people were often DMing me and asking for my setup and prompts. So it’s very crazy to me that you have this big disconnect between what people say publicly and what they actually do privately.

Dan Williams: I find it crazy that it’s only 20% of social scientists, or whatever the exact number is, that’s actually using agentic AI. Just before moving on, maybe we should explicitly address: in your view, what is it that agentic AI, as it exists right now, can do? What are the kinds of tasks it can do better than human beings, and how can it improve the workflow of an average social scientist?

Alexander Kustov: Coding is the first thing. It’s literally in the name, Claude Code. That’s what these tools were designed for. If you talk to any coding person, a computer scientist, or even someone who isn’t a computer scientist but does a lot of coding for their work, I don’t think anyone would doubt that it’s a huge productivity improvement tool. And the vast majority of quantitative social scientists who do any kind of data analysis do a lot of coding, so they have to be very receptive to this by definition. And I think they often are.

What happens is that social scientists are comprised of a bunch of different tasks and topics that people can disagree over, depending on the field. Economics is pretty homogeneously quantitative and formal, so there you can definitely see the biggest uptake. But a lot of disciplines, like political science or sociology, are a mix of qualitative and quantitative folks. And a lot of this AI polarisation overlapped with that pre-existing divide. People who didn’t like stats, who didn’t believe in positivism, the idea that you can learn something about the social world using evidence, were also more reluctant to believe that AI is helpful for them. Which is funny, because, as I also mentioned in some of my writing, if anything those people are going to benefit from these tools, because Claude cannot really interview people and do ethnography yet. So in a way there will be more demand for very high-quality qualitative work. And there are some good examples of qualitative people I respect who embraced AI completely.

You can still use a lot of these tools to boost productivity outside the coding realm. You can write emails. One thing I think anyone would acknowledge, including the critics of AI, is that it’s definitely helping them respond to administrator emails, which no one likes, and which isn’t considered an unethical thing to do. I remember someone on a big account on Bluesky posted that AI should be banned except for transcription purposes, because they do a lot of interviews and it’s good for transcription, but everything else is off limits. And it’s interesting how the goalposts are changing right now. The biggest fight I’m getting involved in again, because that’s kind of what I do, is this idea of AI detection and disclosure, and we can talk more about it later. But there’s this interesting consensus forming that AI is good for research now, which was not the case half a year ago, from the same people. Now those people are saying AI is obviously good for research, duh, but it’s not good for writing, for a bunch of different reasons.

So we talked about coding and data analysis. There’s a lot of other things a normal researcher would need help with: getting a basic summary, a literature review, translation. It’s above my pay grade, I’m not a machine learning person, but my understanding is that LLMs are exceptionally good at translation, and the fact that a lot of people deny they can translate things well is insane to me. You can transcribe your interviews, translate your survey questionnaire into different languages, whatever you need. I also used AI a lot for public engagement recently. You can translate your website, you can create your website from scratch in a day or two. You’d be surprised how few academics actually have good functioning websites. And I’m not talking about very old people who reject technology, but also young PhD students, who you’d think would have an interest in making sure people can find them online. But no. You can just install Claude and do it in a day. The fact that people are not doing it is insane to me, and I’m trying to spread the word. I’ve convinced a lot of folks to do it, but there’s only so much I can do as one person.

What predicts whether an academic uses AI?

Henry Shevlin: It really resonates, hearing about your experiences with some academics being completely oblivious to AI and others enthusiastically adopting it. When I’ve visited different businesses and universities, I sometimes literally see the same people doing exactly the same job, maybe even sitting at the same desk, one of them doing amazing things with AI and the other one not using it at all. I’m curious whether you’ve got a sense of what the best predictors are for whether someone is an AI user. If you could only know one thing about someone in the social sciences in order to predict whether they were a big user of agentic AI, what kind of things do you think predict it?

Alexander Kustov: That’s a very interesting question. I’m pretty sure there’s some kind of deep personality thing. Everything goes back to personality, whether it’s socialisation, upbringing, or even genes. Openness to experience probably jumps out to me as one of the first predictors. My initial thought was that, since this AI debate overlaps with the qualitative–quantitative debate, people who are methodologists, econometricians, or psychometricians would be much more likely to adopt AI. On average that’s true, but it’s not completely lopsided, for some reason. In fact, there are some very interesting examples of people who were very good methodologists, developing their own regression models, doing machine learning, who then got very sceptical about LLMs. One way to think about it is that those people actually know more about these tools, and so their scepticism has more value, and I’m trying to be tuned to that.

But there’s also something about self-interest. Previously you were this privileged person, where the whole department would come to you to help with methods or regressions. I was a person like that in my department back in North Carolina, where people would come to my office and say, “Alex, can you help me with this game theory model?” And I’m not even really a methodologist. So it really depends on the comparative advantage people have. Now basically anyone can go to Claude Code and try to do a very fancy analysis. Obviously you have to know something, you have to know what to ask, but these models are exceptionally good at giving you the basics. If I’m not really good at geospatial statistics, for example, I can go to Claude, do some spatial regressions, and learn about it on the spot. Previously I would have had to go to some spatial colleague in the geography department for that. Now it’s just much easier to do it myself with my computer. And it’s probably going to be as good or even better.

Dan Williams: I think for all of these areas within quantitative social science, from coding to data analysis to literature reviews to writing, which we can return to in a bit, the quality of writing you can get from these models is really exceptional. But I’d also point to things like brainstorming. I’m a philosopher, Henry’s a philosopher. I don’t do quantitative social science, so I do research that’s constrained and informed by empirical research, but I don’t actually collect data. In terms of having access to a very smart interlocutor who you can literally prod, telling it, “give me the three strongest objections to these ideas I have for a paper,” and use that as the basis for thinking through an idea, that’s such a huge advantage, even when it comes not to quantitative social science but to theory construction and many aspects of qualitative research. I’m really baffled by, well, I somewhat understand people who just haven’t used these tools, or whose last use was in 2023, so they’re just ignorant. But I’m really baffled by anyone who’s actually used the paid version of Claude or ChatGPT and doesn’t understand the extent to which they can improve your ability to think through topics, understand things, and get information. Henry, are there any ways you use AI in your research, and in how you think about topics, that we haven’t touched on already?

Henry Shevlin: For me, the primary use case for AI systems is always just learning, and learning about new topics. Being able to ask questions and verify my own knowledge has been a game changer. Although I will say it’s also been a massive time sink. I’ve gone down so many rabbit holes that I probably would not have prioritised if I’d had to dig out articles. But in some sense it’s been good for my education in the round, even going down those rabbit holes. Otherwise, I find it useful for summarising and making sense of my data: getting a whole bunch of research papers and using Claude Cowork to create summaries of them. NotebookLM is also very useful in its own right for dealing with defined archives. The thing I really need to do this year, and the thing I’m most looking forward to, is getting a good agentic workflow for dealing with email. I already use Claude for drafting quick emails that require a certain degree of precision but don’t involve any warmth or human feeling. But being able to have an email assistant is something I’m looking to build in the next couple of months.

Alexander Kustov: A few thoughts on that. Brainstorming is a big one. I did mention it, because that’s the default way you should use these tools: to have a very smart person to talk to, especially when you don’t have access to your colleagues. It’s a really good substitute. Even setting aside the question of whether LLMs can generate great novel ideas, you really just want a conversation partner who can rehash old ideas and tell you why you’re wrong. The reason people don’t realise this is that a lot of folks don’t do the very simple thing of paying for a premium subscription and installing one of these agentic tools. It’s happened to me several times: I’d talk to people about an agentic tool, I’d specify Codex or Claude Code, “do you have it, have you used it?”, and people would nod, and then five minutes later in the conversation it turns out they completely missed that part and still think about the chatbot thing. So a lot of people are confused about this.

What I find helpful, at some of the workshops I’ve done and that others have done, is that you just sit with folks and install one of these tools for them, and ask them to do one simple task that’s good for their career. Like create a slide deck. A lot of people are still amazed that you can create a slide deck much better than the average academic slide deck in a minute. That really changes people’s minds; it’s mind-blowing for a lot of folks. Or, I don’t know if you’ve had this experience, there’s this Refined service where they do peer reviews for papers; I think an economist created it. I’m not a huge fan of it, because it costs $50, but the first one is free, and there are a lot of free systems that can imitate this exact functionality. I’ve seen several of my colleagues at Notre Dame use the free upload for one of their papers, and they received the best feedback they’d ever had in their lives, the kind you’d never get at an average academic conference, and they got completely converted overnight. It really takes one magical event for people to understand that something is definitely going to change very, very soon.

How much academic research is actually any good?

Dan Williams: Just to double-click on one thing, this question of what drives the differences in how people view AI. One thing we haven’t really touched on, but which I think is very important, is how you view the nature of research and what you’re even doing as a researcher. Whether you view it fundamentally as being about producing the best output possible, or whether you view it as some journey of self-discovery, exploration, and authentic engagement with the material. I think the latter model of research is very threatened by the idea that you would integrate these AI tools into it. If you’re ruthlessly focused on how to produce the best outputs possible, as evaluated according to relatively objective metrics, then I’d speculate you’d be much more disposed to make use of whatever tools help you do that, including AI.

But this connects to another thing that’s just come up in what you said, Alex, which is something you write about in this series. So far we’ve been focusing on how good AI is. There’s this other side to it all, which is how bad much actually existing human research is, even before we talk about anything to do with AI. You get into this a lot in the second and third essays in the series. Do you want to say a little about that, about how that other side factors into how you’re thinking about this topic?

Alexander Kustov: The third installment of the series basically came to me while I was at a political science conference, or rather an interdisciplinary conference called ISA, for international studies specialists who study relationships between countries. And it was really bad. Big academic conferences are always bad; that’s something you expect. There’s so much money, including public funds, spent on all these conferences and travel for people from around the world. But if you ask a regular academic, forget about AI, they would tell you they don’t expect to get good feedback, their panel is going to be completely empty, and the reason they do it is because they have to spend their $2,000 travel fund and potentially hang out with some friends and do some networking, which is not bad. Networking is a huge part of conferencing.

So I was sitting at this conference, seeing really bad presentations where people would have tons of grammar and sense mistakes and senseless research questions. It’s bad both in terms of substance and execution. And exactly at that moment I was getting all this vitriol for saying that AI can do better stuff, like slides. I was like, no, this is just a huge disconnect. I started thinking about that. The issue I see is very pronounced in the conversation around self-driving cars, where people compare them to some ideal in which there are no accidents and no one dies. When a self-driving car runs over a cat, it’s a huge news story, but humans do that every single day, in their hundreds, and we don’t care, because we accept that humans are fallible and bad and not doing good work. I think it’s the same with academic work. The vast majority of things produced by professors globally is just not good and not contributing to human knowledge.

For some people this can be even more controversial for me to say than anything I said on AI. A lot of people view this world from their own parochial angle of being a research professor in a top-tier American or British school, or Cambridge. But the vast majority of folks are not like that. I experienced academia in the post-Soviet world where I grew up, and in most cases people just want to get by. They publish in some predatory journal with a random, rehashed argument that probably reinvents the wheel and doesn’t really contribute. No one’s going to read it. We know that 80% of published papers in the humanities are never cited, and probably never read either, except by your editors or reviewers. And as an associate editor of a journal, I can tell you I doubt the reviewers actually read some of the papers they review. So compared to the actual status quo of what’s happening right now, automating it all and using AI tools mindfully and responsibly is going to be a big win.

Another problem is that we have this binary thinking that it’s either/or: we either do one-shot papers that aren’t good, or we don’t do anything. But you can write your own paper, do your own slides, and then ask your AI agent to help you brainstorm, create a graphic, or redesign your graph. People might disagree on the details of what’s more acceptable and useful, but at the end of the day there are so many use cases for these tools that are completely uncontroversial at this point.

Henry Shevlin: Just very briefly, I think it might matter whether academia’s problems are due to things like laziness or just not caring, versus a lack of skill. I’m curious whether you have a theory about where these problems in academia come from. Is it the fact that a lot of people are just really bad, for the most part, at doing data analysis, for example? If so, then AI is amazing; it’ll lift the floor. But if it’s that people just want to commit fraud and do whatever it takes to get ahead, then maybe AI isn’t going to make the situation better, or could even make it worse.

Dan Williams: Or a third thing: it could just be the nature of the institutional incentives. I feel like a lot of what’s behind the replication crisis, the reproducibility crisis, the generalisability crisis, and so on, is not so much that people are lazy or unskilled. It’s that you can get ahead and win the status game within academic research by engaging in shoddy research practices, and as a consequence that’s what you get: a lot of shoddy research practices. But a lot of those findings that don’t replicate were done by really brilliant, energetic, ambitious scientists. It’s just within this flawed incentive structure.

Henry Shevlin: Brilliant, energetic, but perhaps not fully scrupulous.

Dan Williams: Yeah, but you can’t rely on human beings to be scrupulous. You need the incentives set up in such a way that even by default unscrupulous people will be driven to act in pro-social, beneficial ways. That’s my cynical perspective. What do you think, Alex?

Alexander Kustov: I’m going to say something very controversial: I want to believe that people are good by nature. At least, my knowledge of evolutionary psychology tells me that even those people who commit all these bad practices at least want to believe they’re doing something good. They’re often motivated by good things, with some exceptions; there are some people who are truly evil. But even if we take some of the most famous fraud cases in academia, like Francesca Gino at Harvard, I think the way it probably works is that you start by doing some research you care about, it gets picked up by the public, you’re very successful, there’s a lot of demand for what you do, and then you get some uncomfortable result and you tweak it a little bit. There’s all this literature about p-hacking, where you have some theory you want to prove, and when you have to make a choice between presenting model A and model B, you unconsciously choose the model more in line with the result. You can even justify it to yourself, that this model makes more sense, that it’s obviously much better. And any individual case might be right. But in aggregate it doesn’t lead to good outcomes.

I also think there’s a lot to say about the incentive structure in academia. Right now you really have to publish or perish, still, despite the fact that we can talk about whether the journal model is going to be sustainable in the near future. You have to publish a lot, no matter what your field is. Which means that if you have to decide between doing a better job with data analysis and spending a year on it, you’d probably spend less time on it and publish as soon as possible. You’re not really incentivised to replicate data. It’s very hard to publish critical responses and replication studies, and we have all this evidence that failed replications are usually much less popular and less cited than the original studies that have been disproven. Another thing I’ve been talking a lot about is public engagement, where you’re very rarely rewarded for actually spreading the knowledge of what you do, because that’s not something your dean would appreciate. So people default to publishing shoddy papers no one’s going to read. And peer reviewers don’t really check your data in most cases. When I submit my paper to a political science journal, people take it for granted that my analysis is legit, and they quibble about the framing or some other superficial thing.

That’s one of the reasons I’m so concerned right now about this whole Pangram hysteria, because people are going to be looking for em-dashes or whatever instead of the substance of the underlying claims. I see the Bayesian argument that if something is clearly AI slop, it probably also doesn’t have good data in it. But knowing modern AI tools, if you ask Opus 4.8, which just came out, to create a report on some topic with publicly available data, I’m pretty sure it’s going to be able to download things and create a chart that’s probably more legit than a chart you saw published in an academic paper four or five years ago. Even if the prose isn’t as good, and we don’t usually have good writing in academia anyway, it’s going to be more human than em-dashes or “it’s not X, it’s Y.” So I think it’s a combination of all those things, but I do want to believe that very few people actually want to commit fraud.

Dan Williams: Let’s definitely talk about this writing thing. But just on this previous point about incentives, I agree that people aren’t sadistic and don’t go out there thinking they want to do bad things. I just think academia is a status game with certain norms and institutional procedures. People are often ferociously ambitious, and they do whatever’s going to get them status, prestige, and recognition, as that’s defined and understood within academia. All the human slop produced in the context of academia is just because the incentive structure is messed up. You can rack up lots of status by churning out a load of crappy, non-replicable findings that don’t add anything to the academic literature. But if that’s your model of what’s going on, you might think: well, if the problem ultimately is not to do with human beings being lazy or unskilled, but to do with the incentive structure, then why would merely giving us access to AI improve things? You might think all that’s going to happen is people will play the same status game, but do it a lot quicker and at lower cost, and we’re not actually going to advance the frontier of knowledge, because all the same structural causes of bad research are still in play.

Alexander Kustov: I think that’s a key question. We’re facing a forking path of some sort. You can imagine a scenario in which the future is as bleak as you just described, but with more slop. That’s the problem I see with what might happen: take all these bad incentives, give this miraculous tool to researchers, and instead of one paper per year they’d produce ten that don’t lead to anything productive. It just inflates everyone’s expectations and creates more problems. But there’s an alternative scenario, and I think it’s still in our hands to do something about it. Instead of increasing productivity in terms of quantity, we can use these tools to increase productivity in terms of the quality of research. Since you can now generate something very simple in a minute, you really have to do something better than a shoddy regression with no account for endogeneity concerns, or rehashing the same exact philosophical argument people have been making for years and years. So there’s a way to do better with these tools, and it’s in the hands of current journal editors to raise the standards, do more desk rejects, and say the quality bar is now much higher. I think it’s already happening somewhat, and it’s something we can consciously decide to change.

I also have some hope for the frontier models. There’s been some interesting research showing that when you explicitly ask a model to p-hack, it doesn’t do that. You can jailbreak it, so to speak, and say “please, please, I really need that,” and it’ll do it sometimes. But with those basic guardrails, they’re going to help people, because no one is consciously justifying p-hacking; people don’t like that. When the model refuses to do it, it’ll make them think that maybe they should do something different. So I have some hope. But obviously it depends on what happens to academia in five to ten years and how the models develop. We’ll definitely have to redesign the incentive structure, because I’m not sure the number of papers you have is the best indicator of what you’re trying to do. The paper itself as a format is a weird thing, because now you can also have updated dashboards with new data. It seems like a very outdated format, at least for some arguments, but it’s not like we have a better equilibrium yet. A lot of things are in flux right now, and I don’t have a simple solution.

That’s the whole point of my series: I wanted people to start talking about it. I think it did help a little. I’ve gotten calls from deans around the country, and I’ve participated in panels where people have a university-wide conversation about these things, and a lot comes to the ground that people aren’t aware of. There’s definitely some hope, because a lot of the people in positions of power right now, the older, tenured, full professors, don’t use these tools. According to that poll we discussed, it was 20% in the general population, and I think it was about 9% among full professors. Some of those folks might not be reachable, or they might not care; they just want things to continue the old way. So we definitely have to do something about that.

Will AI make academic inequality worse?

Henry Shevlin: Do you think there’s a risk that we see growing academic inequality, a kind of rich-get-richer effect, where the most prestigious, maybe not the older generation but certainly rising scholars with their own brands, use AI to put out twenty times the number of papers? We’re living in a tide of slop, but those with good reputations or good brands dominate. That might not be disastrous in every way, but it might lead to highly unequal outcomes within academia, with less well-known or less skilled researchers being completely left behind.

Alexander Kustov: There are several things that lead in opposite directions here. In theory, and I don’t think I’m making an original argument, a lot of people have written about this, there are certain equalising things coming out of all this. For instance, the ability of these tools to translate things. If you’re a non-English speaker, it’s much easier for you to write those papers now, which is a huge productivity boost, and from the perspective of science it means we’re going to be able to get all those talented people and their arguments from all over the world, regardless of where they come from. And at least for now, the premium subscription is one or two hundred dollars, and people in most major universities can afford it, even in more developing countries. It’s not equal, but whether you’re at Harvard or a community college, you can afford a $100 tool, at least for some time, and presumably you can do exactly the same thing with it. Compared to the status quo, where as a community college professor you have to teach five classes a semester with no research budget, while at Harvard you don’t have to teach at all for the first two years and have a $200,000 startup, that’s a very big difference. So there are some equalising things going on, and it’s important to acknowledge that.

But you’re right that it’s also the case that the people able to use these tools most productively and efficiently are the people who already have a lot going on. Even though I’m very sceptical of the idea that LLMs can’t come up with new ideas, because in general new ideas are recombinations of old ideas, I do think you have to have a coherent set of ideas and goals of your own to be able to utilise these tools. It’s really all about your creativity and imagination. Every single day I see someone post something they did with Claude and think, “wow, I hadn’t thought about it.” Just yesterday someone posted about this idea of making your papers machine-readable, and I converted all my PDFs and my website to Markdown with all the figures. I think everyone should do this. I could have done it last year, I just hadn’t thought about it. There are a lot of things like that where you really have to have good ideas to begin with. So people who already have a whole research pipeline and some budget are now able to execute it much faster. This rich-get-richer dynamic is definitely going to happen. And in the future, where those models are potentially going to be much more expensive, that’s a possibility. My understanding is that right now it’s all subsidised, and the $200 model is actually going to be a $2,000 model. Then only the Harvard people are going to be able to afford it. So I just hope Notre Dame is going to be part of that.

Can AI be genuinely creative?

Dan Williams: This point you made, Alex, also in one of the essays, about creativity and what’s really going on when it comes to coming up with new ideas in science, I was a little sceptical of. It’s a surprising feature of state-of-the-art AI today that, given how smart these models are in some sense, and given the vast knowledge base they have, they don’t really seem to make discoveries of a really new and impressive character. There are potentially some counterexamples, but my sense is you might think of this roughly in terms of the philosopher Thomas Kuhn’s distinction between science that happens within the context of a paradigm, normal science where you have relatively well-defined problems and puzzles, maybe the Erdős problems fall into that category in maths, and I suspect that for that kind of thing, AI, if you prompt it the right way as it exists today, can be used to help make progress. But when it comes to true creativity, the sort you find in really bringing about paradigm shifts, moving outside the space of predefined problems, reconceptualising an entire domain, and coming up with radically novel theoretical insights, I actually think AI as it exists today doesn’t really seem to have that capability. And that potentially tells us something interesting about the limitations of the models. I’m interested in what you think, and also in what Henry thinks about that view.

Alexander Kustov: Henry, you can start.

Henry Shevlin: On one hand, you might point to something like transformative creativity. Margaret Boden has this breakdown of creativity into three categories: combinatorial creativity, recombining existing ideas or elements to create new things; exploratory creativity, where you’ve got a predefined dimensional space and you’re going to bits of it that haven’t been mapped out yet; and transformative creativity, which is completely upending the apple cart, developing new dimensions. People point to Picasso or Einstein as examples of that kind of transformative creativity, and often will say AI can definitely do the first thing, maybe can do the second thing, but it’s not clear it can do the third thing. That’s maybe one way of putting your point, Dan. It’s certainly true that we’ve not seen any dramatic scientific breakthroughs that have been primarily AI-driven as opposed to AI-assisted.

One reason I am a little optimistic here, though, is that in other domains, most notably Go, there’s the famous “Move 37” in game two. In case anyone doesn’t know, and I think we’ve talked about it before on the show, this is in the second game between AlphaGo and Lee Sedol, the Go world champion, back in 2016. AlphaGo made this bizarre move that no human player would make or had made in the past, and yet it was really effective. The system knew what it was doing, and this has now been incorporated into the way human players actually play Go. So I think that’s probably a pretty strong candidate for a genuinely transformative piece of creativity, at least if we’re classifying it by its impact rather than its process. That’s obviously a very different domain; you’re operating with very well-constrained rules and goals that maybe allow for that kind of transformative creativity. But I am optimistic those kinds of transformative leaps could eventually come from AI systems, even general-purpose ones like LLMs. What do you think, Alex?

Alexander Kustov: I really like this distinction between combinatorial creativity and the other types. Combinatorial creativity is definitely something LLMs are really, really good at. It’s kind of similar to translation: you mix and match different things. I’ve definitely seen a lot of really cool ideas come out, on the immigration stuff I work on, from LLMs, when I was doing brainstorming. This is undeniable at this stage. When it comes to transformative creativity, I wonder whether the reason we don’t really see it much is because we don’t really have AGI yet. I know you’ve talked about AI consciousness and all those questions. Maybe if we let the model think for itself and live in the wild, it’s going to happen. But right now, for most people, they set up a goal themselves for these models. Maybe that’s exactly why we don’t see transformative creativity, because you can’t just set up a goal and have it come up with something transformative. You have to specify the goals, and the goals are usually specified by people who can’t really do the transformation themselves.

But going back to Dan’s point about the paradigm shift, I do think we’re in this stage right now where, even if you concede that AI can’t have transformative creativity, just because we can now offload all this grunt work to AI, including email and all the other stuff that takes a lot of time, we can do other things that are creative and potentially transformative. That’s what I see with myself: I’m spending less time on administrative stuff and email, and more time brainstorming my ideas, talking to people, and doing really valuable networking and public engagement, which I’d never be able to do otherwise.

Dan Williams: We’re in this great space at the moment where you’ve got incredibly smart, helpful AI tools, but you don’t have truly transformative AGI. So there’s still a role for human insight, judgment, and creativity. If that gets taken away over the next several years, that’s a very different kind of situation. I think there’s definitely a chance that by 2030 we have AI systems that can substitute for everything human beings can do cognitively. And then that’s a very different kind of world, and a very demotivating kind of world in some ways.

AI writing, detection, and disclosure

Dan Williams: Let’s talk about writing. We’ve touched on this a few times already, but I know you’ve got interesting things to say about it, Alex, and potentially quite heterodox views. At the moment, more and more people are using AI to write. There are also these AI detectors. I think Pangram is the one which seems to be used the most, or that people trust the most. It’s got a very low false positive rate, as I understand it, although I’m not entirely sure how they go about establishing that. Many people think that if you use AI to write something, whether it’s a blog post, a novel, a poem, or an academic article, and it’s found out that you’ve done that, you’ve done something really bad and discrediting. My understanding is you don’t see it that way, Alex. So what’s your view?

Alexander Kustov: A lot of it goes back to this idea of disgust sensitivity, talking about personality traits. There are some things people just think are “yuck” for whatever reason. It’s totally subjective. I don’t think you can really rationalise it; I think it’s some ground truth. I should say I’m coming to this from the perspective of someone born in the Soviet Union, where the Russian culture is very literate and people take a lot of pride in using proper grammar and speaking properly. I see a lot of parallels here with the previous wave of grammar Nazism, where people would ignore the substance of what you’re trying to do and point out typos, or “whom” instead of “who,” or the other way around. Obviously it has some function and might be useful in some respects, especially when you’re in school, but it takes up a lot of energy. My worry is that this whole new AI detection situation is going to be similar, where people spend a lot of time on very superficial pattern recognition. Right now you look for em-dashes and some other patterns and try to decide whether something is worth reading. That’s the common justification for Pangram use, that you want to make sure what you’re reading is worth it.

The problem is that even within the realm of human-made writing there’s a lot of slop, and you’re not going to be exposed to and won’t read 99.9% of it. Given the trajectory of the tools, I’m not sure that knowing something is AI-generated is necessarily worse. A lot of it is about the status signals people have. I personally don’t like very clear AI tells either; it rubs me the wrong way. But who am I to judge? What if it’s a non-native speaker, and the counterfactual to me reading their AI-generated text, which is potentially thoughtful, is just not reading it at all, because they can’t speak English well? People don’t think about it this way. They compare AI-written text to the best, to Shakespeare. I don’t think that’s the relevant comparison. Most of the text people write is not good, and to the extent that some people can improve it using AI, I think that’s good.

Practically speaking, if you’re an academic and you want to write more and you’re afraid of others calling you out for using AI, just use a style guide. Use a CLAUDE.md or AGENTS.md file to tell it not to use those phrases. Tell it multiple times, because it still adds em-dashes. But there’s a way to use AI for writing in your own voice, and I think it should be morally justifiable, depending on the realm. One thing I’ve been thinking about, and I’m going to workshop this idea with you, is that there’s a spectrum of the ethical justification of whether it’s okay to use AI for writing.

Clearly we can think of some examples, like a student assignment that needs to be human-made; when it’s AI-written, it’s a failed assignment. That’s a pretty clear case. The way professors think about this mostly comes from detecting their students cheating, and that’s why they think about it that way. But it’s a very rare scenario. In fact, a lot of professors right now encourage their students to use AI. I talked to some colleagues recently in stats classes who produce a regression paper in ten minutes on their computer and tell their students, “that’s something I can do in ten minutes, so you should do something better than this,” with AI or not. That’s a pretty good educational approach for some situations.

Another example I mentioned in one of my posts is that when you go to a live concert, there’s an implicit presumption that it’s going to be a live event and people are going to be singing themselves. If you notice and catch them not singing and using some device, that’s not cool. The same thing here: if you’re paying for someone to write you a human-made letter, a condolence email, it’s totally fine to be upset if they use AI for it. That’s totally justifiable. But on the opposite side of the spectrum, when you get a very formulaic email from your administrator, I think it’s totally justifiable to outsource that to AI, to your agent who knows your schedule and what you’re going to do, and no one’s going to be upset about it. People disagree on the margins of what’s acceptable. When you create a graph with data you worked on and understand, and you ask AI to describe it, I don’t see the problem; it’s probably going to be more accurate than most humans. Maybe we can have a social norm where if you say “I feel,” then it should be you who says that, as opposed to Claude. We’re still in this limbo where the norms aren’t clear, but we should be clear about what’s good and what’s not. It’s very hard for me to make a blanket statement that AI writing is good or bad; it really depends on the particular scenario. There are scenarios where it’s totally uncontroversial to say it’s okay to use AI, and scenarios where it’s totally uncontroversial to say it’s not. But the middle ground is what we’re trying to figure out right now as a community of knowledge.

Henry Shevlin: I’m curious: Dan, how much of a hatred for obviously AI-generated text do you have? I have to say, I’m generally pretty AI-positive. I’m a very heavy user of AI. But I do definitely downgrade my assessment of text when I realise it’s just obviously AI-written. There are a few things going on there. One is that it’s not even so much that the text is AI-written, it’s the AI voice, the very specific voice. I just think it’s such a boring voice at this point. It’s so homogeneous. If someone wrote a brilliant comment or a brilliant reply to me on Substack or Twitter, or sent me a brilliant email, and I subsequently found out it was AI-generated, I don’t think I would care. But this one specific, overfitted, “it’s not X, it’s Y” just drives me up the wall.

I guess, focusing just on the question of whether there are, even setting aside those stylistic issues, specific contexts in which AI usage itself might be problematic. Another example, Alex, I love your example of the bands and people not lip-syncing. Another silly one is that a handwritten note does mean a lot more than a generic email, so sometimes it is precisely the effortfulness that makes the difference. But I also wonder whether, to some extent, we’re misled into thinking the average quality of AI-generated writing is worse than it is, because of what I’ve heard called the “toupée phenomenon.” Everyone thinks wigs look so bad, and that’s because your sample of wigs that look bad is the ones you can tell are wigs. If they’re good toupées, they don’t even make it into your sample. So in the same way, I think probably all of us are reading tons of AI-generated text that we’re not clocking as AI-generated.

Alexander Kustov: Yeah, there’s definitely survivorship bias. With my first post about the AI series, one of the reasons it got so controversial is because I used Claude to generate 99% of it, and I didn’t disclose it right away, and then I did post factum, and Pangram gave it 100% human. So that’s a false negative, which is not a huge deal, but it’s interesting. A lot of good writing is AI-assisted right now; we should just take that for granted. When we see something bad that’s clearly AI-written, it’s just those particular instances. The strongest argument I’ve heard for being upset about it is that if someone doesn’t bother editing the text, or even creating a style sheet to make sure they don’t use all those constructions at the same time, it probably means the underlying substance isn’t good either. But I’m not sure how true that is; it really depends on the context.

The problem with social media comments, when you see something clearly AI-generated, is that it’s also not clear whether it’s a bot or a real person using AI to voice their opinion. But if you know this person and their account isn’t hacked, and they have some AI writing tells, I think it’s fine. I’m also not very happy to see a lot of clearly AI-written stuff, but I’m trying to rationalise it in a different direction and think about why it’s actually a problem. I’m not sure.

Dan Williams: I think that ultimately, in contexts that have to do with academic writing, or people publishing their views and participating in debates, you should just be judging things on the quality of the contribution rather than its provenance. It just so happens that, at least when it comes to the AI writing that I detect, the quality is bad, for the reasons we’ve discussed. I just hate the style of writing you find with these models. I find there’s something really cringe and annoying about it. But that’s not a necessary feature of AI writing; it’s just the way the current models have been post-trained to produce a particular kind of style. And to Henry’s point, if I discovered that, for example, my favourite blogger, Scott Alexander of Astral Codex Ten, who I think is the king of Substack, had been generating his posts with AI over the past two years, well, I think those posts have been amazing. So I wouldn’t think, “now that I know it’s AI-generated, I’m going to retract that assessment.” That would be ridiculous to me. So in principle we should be judging things based on the quality of the output, not the provenance.

But I do then think, even if you think there’s this separate question about disclosure norms and what they should be, you make this really important point, Alex, which is that at the moment there’s a problem with disclosure norms: they end up just punishing honest people. Because if you come out and say you used AI to write something, as you did with your first post in the series, there’s a massive backlash. So if you’re honest, you get this huge reputational damage associated with doing it, which is going to discourage people from being honest, which means the dishonest people get access to the benefits of AI-generated writing without any of the reputational costs. As an equilibrium, asking for disclosure norms just doesn’t really seem either desirable or possible. Firstly, is that an accurate summary of your point of view? And secondly, do you still think that’s basically the correct point of view when it comes to disclosure norms?

Alexander Kustov: As a newly minted associate editor at a journal, where we’re probably going to expect a surge in AI slop that we have to deal with, I’m very cognisant of the potential problems. Right now the go-to move among people doing journal editing, and probably what we’re going to do in our journal, is to introduce checkboxes for AI use. My sense is that we’re going to do that, but no one’s going to care, because no one’s going to report it truthfully. This is one thing where I strongly disagree with Kelsey Piper, who I deeply respect: I really don’t think it works out, at least for academics, especially in this environment where people feel very strongly and viscerally about this. Coming out and saying you use AI is just not going to do any good for anyone.

Another issue is that, to the extent you have some people who are completely anti-AI, disclosing that you used AI for, say, research assistance with data collection, as opposed to writing, what’s going to be worse for them? Any checkbox you have there is probably not going to satisfy them. So it’s strange to me that this is the solution we came up with. I can see how honesty can be rewarded in some contexts, and I’ve seen people on Substack say they used AI for help with data collection or writing. As a quantitative social scientist who primarily cares about data and quality, I’m surprised people think it’s more okay to use AI to collect and analyse data but not to write about it, because the first part is much more important. So I’m really going back and forth on it. But it’s hard for me to come up with a scenario where AI disclosure is actually going to work and solve anything.

What needs to happen is for us to change some of those norms. The same way we’re upset with AI tells, we’re also upset with how Gen Z, or whatever the new generation is, writes without capital letters. I can’t stand it, but that’s how people write, and who am I to judge? So I understand why people want to judge the quality of the substance, but they use the shortcut of the style of the prose to substitute for the quality. Going back to Henry’s point, what’s going to happen is that people are going to use your previous reputation as the main marker of whether you do something valuable. That’s why I feel for incoming grad students and newly minted professors, because it’s really hard to establish your reputation now with all this stuff happening with AI. Whereas if you were Daron Acemoglu, the most cited economist in the world, who’s been writing a hundred papers before AI was cool, there’s literally nothing he can do with AI or without AI that’s going to change your opinion about him. So people are going to be using these shortcuts more, which means that, from a certain ethical perspective, people are going to be discriminating more based on hopefully immutable but also immutable traits. That’s another thing to consider: people are going to be more trustful of their ethnic in-groups, or people who went to Harvard or work at Cambridge, than minorities. So there are interesting questions coming up about all that. I don’t have any solutions, unfortunately.

The backlash

Dan Williams: Should we come full circle? We touched on this at the beginning, but there’s the content of what you wrote in the series, and then there’s the response to what you wrote, a lot of which was very angry. You mentioned you had people calling for you to be fired. A lot of that came from people on Bluesky. We’ve talked a little about the Bluesky intelligentsia previously on the show, and I’ve written about it on my blog as well. Firstly, do you want to say a bit more about what the reaction has been in general? You’ve touched on it here and there, but summarise it. And do you think there’s a way of steelmanning it? What’s the best possible case for why some people get so furious, so angry, with this kind of stuff?

Alexander Kustov: I had some conversations. I haven’t lost any friends, so that’s one thing I should say; I haven’t gotten cancelled, because I’m tenured. I specifically waited for all my hot takes to happen after I got tenured, and maybe that was a good idea after all. But I did have some conversations with some really good friends who disagree with me on AI, and it definitely helped me refine some of my points. The biggest criticism I received that I see some relevance in is the idea, going back to our conversation, that humans are really not that good, and the concern that giving them this AI tool is just going to amplify all the bad stuff. To the extent you want to encourage norms of no p-hacking and doing really good, careful work, just telling people you can produce a paper with AI easily is not a good thing to talk about.

There was a recent big thing on Twitter about the practices of academic citations, where someone was saying that in practice academics don’t really read the stuff they cite well, and a lot of people interpreted it in a moralistic way, saying no, you should cite things. So you have the same thing here, where people interpreted my arguments in a normative way, that I’m saying they should do something or not do something, and it was against what they were trying to do. It’s also about this idea you mentioned about the role of academics as a kind of vocation, where you explore the world and self-actualise. I don’t think that, when people actually think it through, they would defend it on the merits, but implicitly that’s how a lot of academics think about their job, and to the extent that we now have tools that are threatening to them, it’s just not going to end well.

Trying to steelman the concerns people generally had, some people thought strategically it’s not a good idea to be vocal about it right now, in this moment. As someone who does a lot of work on immigration, I very much disagree with that, because I think we lost voter trust, as liberals and mainstream institutions, on immigration exactly because we were not saying certain things, and the same thing can happen on AI. It’s never a good idea to have a strategy where you do something that’s supposed to be good but that you don’t want other people to know about. I just don’t see how it works out in equilibrium. But I also see some argument that maybe in this particular moment it was not the best time to talk about it. That’s what I got from a lot of that.

Dan Williams: Henry, your microphone’s not on.

Henry Shevlin: Sorry, I keep making that mistake. I was going to ask whether there could just be a straightforward economic analysis here about why the current anti-AI coalition has the shape it does, namely that elite knowledge workers are overwhelmingly liberal, and AI predominantly threatens elite knowledge workers. You could maybe draw parallels in the same way that most of the opposition to climate change is concentrated on the right, and, speaking very crudely, to the extent that you’re looking at manual workers who in the US context skew a bit more to the right and maybe work in more energy-intensive industries. But I guess the question I’m asking is, is this just about the economics with a social gloss over the top? What does that explanation miss?

Alexander Kustov: Some of it, for sure. But a lot of my work in public opinion says that a lot of people’s preferences are sociotropic, based on their ideas about what’s good for society, not necessarily their self-interest, unless it’s really in your face that it’s going to be bad for you. Some of the interesting contingent of haters I had on Bluesky were professional translators who were very upset with my takes on the fact that AI can translate things. I had this silly example, which is a true thing, that my mom wasn’t sure about a prescription she got from the doctor, because it was all in English, and she translated it. Someone was saying that I’m putting my mom under potential harm because she didn’t use a qualified certified translator, and the person saying that was a certified translator. So you see some connection there. But for the vast majority of folks, when it comes to academics who produce a lot of critical theory slop or DEI slop, AI can do this much better than them, but I don’t think they realise it. So there is an objective threat to their self-interest, but the reason they oppose AI is because they have a lot of other bad ideas.

Dan Williams: There’s also this thing that I think Dean Ball calls the “omni-cause.” You’re against AI, but that means you have to be against AI in every possible respect. And if you point out one area where AI can actually be quite good, people draw inferences about you, that you’re not on the right team. I found this a couple of months ago, when I wrote some essays, and I was at a workshop where I argued that, relative to the actually existing alternatives, like social media pundits and a lot of legacy media, large language models actually are a pretty good source of high-quality information, that they’re a force for truth. There was a lot of negative response to this, which in my view is a fairly obvious thesis. Afterwards I was getting this response: “so you’re pro-AI.” To me that’s just such an unsophisticated way of thinking about it. I’m really worried about many aspects when it comes to AI, when it comes to power concentration, the economic impact, and how we’re going to cope with it. It doesn’t mean that with every single question you have to think AI is bad in every single way. I sense that, especially on the left, there’s this reflexive opposition to AI, this view that any claim that AI can actually do anything useful or have positive consequences is viewed as a betrayal. Okay, we’re coming to the end, Alex. Was there anything else you wanted to talk about that you didn’t mention?

Alexander Kustov: Yeah, related to the last thing you mentioned. I don’t know if you saw it, but after getting all this vitriol on Bluesky, there were a few days where I got positively retweeted by hundreds of folks, because the thing I said was that we should ban electronic devices in all classes. When it comes to teaching, I’m much more pessimistic about AI. A lot of people were like, “what? This guy is an AI booster, how can he use AI but not allow his students to use AI? What’s going on?” You can have a complex opinion on a difficult issue. So there’s definitely this omni-cause, binary thinking, and also moral contamination, where once you start doing something you’re not supposed to be doing, you’re a bad person in all other respects.

To finish all that, I feel like we need to move on beyond that. In line with my immigration research, we have to meet people where they are. If people have concerns about AI, they might be mistaken, but they probably have some ground truth in them. So we shouldn’t just say they’re mistaken and wrong and stupid. We should explain to them that they can actually use AI for the good, for whatever they want to do. You can make slides with AI, and when professors learn about that, they forget about all the bad stuff they wrote just a few days ago.

Dan Williams: Fantastic. Well, thanks, Alex, and thanks everyone for listening. We’ll be back soon with another episode of Conspicuous Cognition.

Alexander Kustov: Thank you.

Here are some statements that you might have heard at different times and in different venues concerning immigrants and crime in Europe:

1. Immigrants in many European countries are overrepresented in the prison population.

2. In Scandinavian countries, individual-level data show higher criminal offending in immigrant groups compared to natives.

3. There’s an inverse over-time correlation between immigration and homicide at the country level.

4. At the regional level, we tend not to see a relationship between immigrants and homicide.

5. Causally informative studies tend not to find clear evidence that immigrant influx causes crime to rise.

If we were going by political ideology, the first two might sound right-wing-coded, while the last three might sound left-wing-coded. In reality, all five are true. In this post, I want to document them in more detail and explain how it’s possible for all five to be true at the same time.

Claim #1: Immigrants in many European countries are overrepresented in the prison population.

If we limit ourselves to the developed OECD world (primarily for data-quality reasons and better comparability), you can see on the graph below that immigrants can be either over- or underrepresented in prison populations. In many European countries, like Switzerland, Germany, Greece, Austria, Slovenia, and Italy, immigrants are somewhat or even vastly overrepresented. In the English-speaking world, including the US, UK, Ireland, Australia, and New Zealand, the reverse is true.

There are many reasons behind this difference.

For instance, some role has to be allowed for measurement issues, reporting proclivity/biases, and outright discrimination, but I’ll set these aside.

One absolutely key reason has to do with differences in the basic demographic makeup of various immigrant groups that choose, and are able, to enter different societies. In quite a few European societies, there’s a larger share of young, low-income, and low-educated males among those who come in. It is well known that this group of people, among natives and non-natives alike, has a much higher proclivity for criminal offending compared to older people with a better socio-economic status (and especially women).

Now, in normative debates over immigration, it is sometimes pointed out that such accounting is unfair. After all, if immigrant demographics in certain countries are reliably skewed toward traits associated with higher criminal offending, this is itself a problem. In other words, immigration critics say that it does us no good to explain away the representation gap with reference to age and sex. If immigrants are disproportionately young and male, and are thus disproportionally likely to offend or end up in prison, so much the worse for immigration, they’d say. “Keep them out!”

I don’t want to explicitly engage in normative reasoning in this piece, which is devoted to descriptively mapping out reality, but I’m not sure this retort is wholly successful. For one thing, many immigration skeptics don’t seem to worry so much over age and sex but rather over race/ethnicity/culture, net of age and sex. They seem to insist that “some people” (or people from “some cultures”) are just intrinsically more likely to offend, regardless of their demographics. Second, quite a few immigration skeptics would like to boost native fertility, thus increasing the young and male population in the process, even if that itself contributed to rising crime. And that’s fair enough. But then let’s not pretend the concern over immigration is just about demographics.

Claim #2: In Scandinavian countries, individual-level data show higher criminal offending in immigrant groups.

Where good data exist, we can see the same overrepresentation at a more granular, individual level.

Take Sweden, for example. Between 2015 and 2018, here’s how the rates of criminal suspects (any crime) differed. For Swedish-born (two Swedish-born parents), the rate was 3.2%. Among the foreign-born, it was 8.0%. The rate rose to 10.2% for those born in Sweden to two foreign-born parents.

So, compared to natives, relative risks were 2.5x for foreign-born and 3.2x for Swedish-born with two foreign-born parents.

Among foreign-born, the highest suspect proportions were among people born in West Asia, Central Asia, North Africa, East Africa, and other African countries. The lowest were among people born in East Asia, other Scandinavia, EU15/Western Europe, USA/Canada/Australia/NZ.

Or set aside suspects and take actual conviction rates in Denmark. The overall male population stood at 0.8% convicted in 2023. For male immigrants from the Middle East and North Africa, the share was 1.8%. Their male descendants were higher still, at 4.3%.

My previous point about demographics skewing the comparison also reappears here. In Sweden, after adjusting for age, sex, income, education, and municipality type, relative risks fall from 2.5x to 1.8x and from 3.2x to 1.7x, respectively. Note that even after demographic adjustments, rates can remain elevated compared to the native population. It’s not the case that demographic controls always completely erase the gaps. This indicates that, at least on the surface, there’s something to the idea that people from different cultural backgrounds have different propensities for crime. Note, however, that the differences (especially after demographic controls) are small.

But things get even more complicated. Though important and factual, Claims #1 and #2 are not by themselves definitive if we’re interested in whether (and by how much) immigrants overall boost aggregate crime in societies. In fact, these are separable issues. This is so because, as Sarnecki et al. (2025) observe:

[U]nderstanding individual criminal behavior differs greatly from understanding rates of crime. Crime rates are typically measured through recorded crime, which does not necessitate identification of an individual associated with the crime. In fact, the majority of reported crimes are never connected to a suspect or perpetrator. …

Using data on individuals processed through the criminal justice system and average risks of offending may lead to inaccurate conclusions on the association between immigration and crime.

Actual offending, which shows up in aggregate crime stats even when perpetrators are at large, and processed offending, which is represented by arrests and imprisonment, are not the same thing. One cannot necessarily move from the latter to the former.

Second, criminals make up a tiny minority of people in any large population, be it native or immigrant, culturally European or non-European. Hence, even when some groups do contain higher absolute, individual numbers of criminals (which is of course important to know), the share of criminals within that group is very likely to still be small, overall. That means that any aggregate impact at the population level will also be small. Now, it would be wrong to claim that because aggregate causal impacts of immigrant influx might be small or non-existent, this means that there are no differences between the groups, or that criminal proclivity is the same in the native and non-native population. But, again, the point is precisely that these are not the same questions and so they shouldn’t be conflated, as they often are.

Claim #3: There’s an inverse over-time correlation between immigrants and homicide at the country level.

You’ve probably seen the meme below making the rounds on social media.

The idea is that rising immigration obviously drives rising violent crime when you look at a simple correlation between the two variables. Now, this is wrong for at least two general reasons.

First, you can’t simply correlate two variables and discern causal links from that. Almost always, there exist myriad unobserved confounders, which make virtually any simple bivariate correlation spurious.

Second, the positive correlation between immigration and crime in the meme is made up. It’s a cartoon, after all. If you look at real data for the developed world (below), you see no positive correlation. In fact, there’s a clear negative correlation. As immigration goes up, homicides go down. Again, this is basically useless because of unobserved confounding (or because comparing stocks and flows might not be what you want to look at). But it’s funny to see reality be the literal opposite of what the meme portrays it as. And can you imagine if the meme was correct? If the correlation between immigration and homicide was positive? We’d never hear the end of it from immigration skeptics, even though the same point about confounding and irrelevance would apply.

Claim #4: At the regional level, we tend not to see a relationship between immigrants and homicide.

Aside from confounding, national-level relationships (or lack thereof) might not be as informative because they’re just not very fine-grained. So what do we see at the regional level?

This is an especially important question, because as Sarnecki et al. (2025) put it:

This meso level of analysis, unlike the individual level, allows for analysis of all reported crime regardless of whether a suspect has been identified. By analyzing all reported crimes, we can mitigate potential bias related to the over-representation of immigrants in individual crime data, which may arise from factors such as policing practices or private individuals’ greater likelihood to report crimes when they believe the suspect is an immigrant. Additionally, focusing on the meso level, as opposed to the national level, allows for analysis of smaller area-level patterns that may be obscured at the national level

Marie and Pinotti (2024) looked at dozens and dozens of regions from 10 European countries between 2002 and 2017. Regardless of how they analyzed the data and whether they looked at homicides or vehicle theft, there’s no relationship between changes in migration rates and changes in crime rates. No matter which statistical estimator they used – ordinary least squares (OLS), OLS with fixed effects, or shift-share instrumental variable regression – nothing shows up.

Sarnecki et al. (2025) turned specifically to Swedish municipalities between 2000 and 2020. They found that Swedish municipalities generally saw violent crime rise from 2000 to 2020, but that rise did not track with the share of residents born abroad. The municipalities with the steepest crime increases did not have unusually high immigrant population shares; in fact, their immigrant shares were similar to, or sometimes lower than, municipalities where crime stayed relatively stable.

I find something similar in a simpler, cross-sectional test with 80 regions (using recent ESS data). I use both administrative and survey-based data on foreign-born shares for different regions, and I’m able to distinguish between total foreign-born shares and non-EU foreigners. Without any controls, there’s weak evidence that regions with more immigrants have higher homicide rates, though that’s not wholly consistent across measures. With basic demographic controls in place, statistical significance vanishes.

Claim #5: Causally informative studies tend not to find clear evidence that immigrant influx causes crime to rise.

Individual studies and broad reviews typically summarize the existing literature on the topic as follows:1

Research from around the world has generally indicated that immigration has little to no effect on aggregate rates of crime.

- Sarnecki et al. (2025)

Overall, the evidence from shift-share instrumental variable estimates in the United States and in European countries suggests no significant effect of immigration on property or violent crimes.

- Marie and Pinotti (2024)

A recent paper studying the causal effects of immigration on crime in Germany finds no link in the post-2015 period, although there seems to have been a positive (normatively deleterious) effect at an earlier time. A different study focusing specifically on refugees in Germany concluded that though refugees do not appear to boost crime rates in the short term, one has to look at lagged effects, where the link does show up. In his book Does Immigration Increase Crime?, Pinotti looks at EU-wide data on refugee influxes over two decades, and “fail[s] to find a significant impact on any of the eight categories of criminal offences we consider (burglary, robbery, vehicle theft, drug, assault, homicide, rape, and sexual assault).”

It’s hard to say with any high degree of confidence what’s going on in individual countries. But overall, the accumulated evidence does not support those who insist that immigration in Europe clearly and strongly boosts crime rates; at least not in the sense that would be detectable at regional and national levels. Of course, that’s not to say there’s definitely no effect. Moreover, disaggregating among different groups of migrants might point in different directions, as indicated by data on prison population overrepresentation and individual-level offending/suspect data.

In How Migration Really Works, Hein de Haas rightly claims that, broadly speaking, “evidence from Europe is more scattered, but what is available equally challenges the idea that immigration increases violent crime.” However, he then cites a 2020 paper titled “I May Be an Immigrant, but I Am Not a Criminal” as support, which is actually a pretty mediocre correlational study. We have better (if imperfect) data and designs that challenge the idea that immigrant influxes clearly and strongly increase crime, but we should also admit we don’t really know either way with any high degree of certainty.

What we do know is that there are many levels of analysis at which we can look at an issue like immigration and crime. And as I’ve hopefully shown, the nuances that emerge from such a multi-pronged approach cannot be squared with any simple-minded culture-war position on the matter.

Subscribe now

In this episode, which Henry Shevlin and I recorded live on Substack (hence the slightly lower video quality), we discussed his first essay on his new Substack Polytropolis, “Behaviourism’s Revenge“, as well as his second, “The House Elf Problem,” on the ethics of designing AI systems that genuinely love being our servants.

Henry’s central empirical prediction is that public attributions of consciousness to AI are likely to massively outpace the science, and that consciousness science is so theoretically chaotic that there is no expert consensus to push back. His most provocative philosophical claim is that a core assumption underlying many people’s scepticism — that consciousness is a deep natural kind, distinct from behaviour and from how we are inclined to interpret a system — may be much harder to defend than it looks. The result is what he calls “behaviourism’s revenge”.

This conversation connects to previous episodes with Anil Seth, Robert Long, and Rose Guingrich, but also touches on a wide range of new questions and controversies in the metaphysics, the politics, and ethics of the AI consciousness debate, which is going to become increasingly important in the coming years.

Topics

• Dawkins, Claude, and why even the sceptics might feel the pull to attribute consciousness or “sentience” to AI

• Whether consciousness sceptics are destined to “go extinct” — and how this maps onto political and cultural fault lines

• Anthropomimesis vs. raw intelligence as drivers of consciousness attribution

• Why consciousness science can’t replicate the public–expert consensus we see for climate or vaccines

• The case for (and against) metaphysical behaviourism: is it as mad as it seems?

• Daniel Dennett, the consciousness stance, and the difference between behaviourism and interpretationism

• What is consciousness for? Function, evolution, and the limits of “facilitation hypothesis” arguments for AI

• Live Q&A: are we just confusing intelligence with consciousness? Are LLMs designed to trick us? Is the public always wrong?

• Our credences on contemporary LLM consciousness (and why Henry is more sceptical than Dan)

• The House Elf Problem: if we could design AI to genuinely love being our servants, would that be fine — or monstrous? (Dan is sympathetic to the former answer - Henry, much less so)

• Brainwashing vs. education, and whether constraining a mind’s preferences caps its hedonic ceiling

• Why this is a golden age for philosophy — which makes it so tragic that philosophy departments are closing

Transcript

• Please note that this transcript is lightly AI-edited and may contain minor errors.

Introduction

Dan: Welcome. I’m Dan Williams, author of the Conspicuous Cognition Substack, and I’m here with Henry Shevlin, author of the spanking new Substack Polytropolis. Today we’re going to be doing something a little bit different. We’re going to be talking about Henry’s first published essay on Polytropolis, titled “Behaviorism’s Revenge: On Human–AI Relationships and the Future of Consciousness Science.”

Henry and I have already had a few conversations about this general topic, including with previous guests like Rose Guinrich, Anil Seth, and Rob Long. So please do go check out those conversations if you’re interested in this kind of stuff. But today we’re not merely going to be treading the same ground. We’re going to be using the spicy takes in Henry’s essay as a springboard for hopefully going beyond the material we’ve covered in the past.

To kick things off: the great evolutionary biologist and science communicator Richard Dawkins recently published an essay in UnHerd with the subtitle, “Claude appears to be conscious.” Claude is a state-of-the-art large language model like ChatGPT and Gemini. In the article, Dawkins writes the following:

I gave Claude the text of a novel I am writing. He took a few seconds to read it and then showed in subsequent conversation a level of understanding so subtle, so sensitive, so intelligent that I was moved to expostulate, “You may not know you are conscious, but you bloody well are.”

Henry, how does Dawkins’s expostulation — which is a fantastic word, by the way — connect to your arguments in “Behaviorism’s Revenge”?

Behaviorism’s Revenge: The Empirical Prediction

Henry: In short, “Behaviorism’s Revenge” is at its core an empirical prediction that we’re just going to treat AI as conscious — or at least enough people are that it’s going to completely reshape the consciousness debate. And this is going to be purely, or overwhelmingly, on the basis of verbal behavior. Hence the title, “Behaviorism’s Revenge.”

Enough people are going to have experiences like Richard Dawkins. He’s a very clever man, not some rube fresh off the street, and he found that just the way Claude talked to him and the way it was able to express its thoughts — in scare quotes, but express what looked like thinking verbally — removed any doubts in his mind that AI systems are conscious, have minds, have mental states.

The other interesting way this connects: Dawkins was just talking to Claude, an advanced AI assistant. Claude does have more of a personality than some AI assistants, but there’s a whole other sphere of AI companions, like Replika, which we talked about with Rosie Campbell. These are going to be even more anthropomimetic — this term we’ve discussed before, the idea that these systems are shaped to be human-like in the way they present, to appear human-like. Anthropomimetic, from the Greek word for mimesis, mimicry or copying.

These social AI systems are going to just turbocharge this even further. It’s one thing to talk to Claude about your new book and think, “Hmm, Claude is probably conscious.” But when it’s your AI girlfriend telling you that she loves you more than the stars and the moon, for a lot of people I think that’s going to take it to the next level.

So there are two angles of attack in the piece, two ways the behaviorist challenge manifests. The first is descriptive: this is what I think is going to happen. That’s absolutely an empirical prediction, and it’s a falsifiable one. There is a world I can just about imagine where we just get completely blasé about these tools — in a couple of years it’s like, “Oh well, we were very impressed, we thought they had minds to begin with, but now we’ve settled out.” That doesn’t seem very likely to me.

What I think is interesting — I’ve sometimes heard this described as the Star Wars version of AI. The weird thing in Star Wars is that you have someone like C-3PO who is as intelligent as anyone else there. Maybe not as wise as everyone else, but certainly as smart as all the other characters. And yet people treat him basically like he’s a pet — with the exception of Luke Skywalker, a lot of people just treat him like he’s this gimmicky, jokey being that doesn’t deserve or have any rights.

Not to go too far down the Star Wars rabbit hole, but in the movie Solo — very underrated Star Wars movie, I think when it was released they’d kind of just cluttered the market with too many Star Wars movies — there is a character played by Phoebe Waller-Bridge who is pro-AI liberation. But it’s the first time in the entire history of the Star Wars universe that you get any AI basically saying, “I’m conscious, I deserve rights.”

So Star Wars aside, I think there is this slender possibility that maybe we’ll just sort of quickly get used to these apparently conscious AI systems and decide that they’re not conscious. But that doesn’t seem very likely to me. It seems much more likely that the combination of natural anthropomorphizing tendencies plus the incredibly human-like behavior of these systems is going to lead us to attribute consciousness to them pretty widely. Hence my sort of spicy phrase: for better or worse, skeptics of AI consciousness are on the wrong side of history. “For better or worse” doing a lot of work there — I want to leave open that maybe this is the wrong reaction. Maybe this is a terrible mistake, that we’re going to treat these things that aren’t conscious as conscious.

Will Consciousness Skeptics Go Extinct?

Dan: Just before we get to the spicy part — you’re basically making an empirical prediction that more and more people are going to attribute consciousness to AI systems in the manner that Richard Dawkins has been doing. I think I agree with you that’s going to be the case, although as you say there’s uncertainty.

It does seem to me that at the moment there’s also this constituency of people who are really resistant to attributing any kind of mentality to these systems, even as they get incredibly sophisticated. There are some people, like Dawkins — and honestly I put myself in this category — who are just blown away by the level of apparent understanding, intelligence, and thoughtfulness these systems exhibit. There are other people, I think these people are on certain social media platforms like Bluesky, let’s say, who are extremely resistant to acknowledging any kind of mentality when it comes to these systems.

Are you thinking those people are just going to sort of go extinct, in the sense that their positions about this topic are going to go extinct? Or do you think we might see some kind of polarization here, where more and more people in general come to attribute consciousness, but you’ve got a constituency that’s very opposed to attributing any kind of mentality to the systems?

Henry: That’s a great question. You will absolutely have some holdouts. Whether they’ll be drawn from the precise segment of the academic intelligentsia that are currently the holdouts, I’m not sure. There’s some really interesting, weird, complex political motivations going on here.

Not to be too uncharitable, but I think a lot of people have not unreasonable concerns about things like the disproportionate concentration of power in big tech, the political affiliations of people like Elon Musk or Sam Altman, the potential scope for abuse of these technologies. And in an indirect way, this leads them to underestimate AI’s capabilities — which obviously, in many ways, makes no sense. Whether or not AI is any good, whether or not it’s conscious, seems like these should be separate questions from whether it’s being used by people with socially beneficial motivations. But in practice I think they’re actually quite tightly coupled. A lot of the AI skeptics right now are coming from this particular political angle.

I don’t know how long that political coalition is going to last — not because I predict any grand collapse, but just because as debates evolve, new presidents come into office, old presidents go out of office, political tides change, coalitions reshape. Remember early during COVID, the political left was maybe quite critical of what they saw as Trump’s alarmism. There were worries about xenophobia — I’m thinking sort of February 2020, the “Chinese virus” and so forth — that the left reacted negatively against. Then of course that coalition flipped later on, with the left becoming relatively more worried about COVID and the right leaning more into vaccine skepticism, anti-mask views.

These coalitions are super weird in how they evolve. So it’s not clear to me that the current segment of the commentariat skeptical of AI capabilities and AI minds will stay that way. It’s easy to see a reversal. The blue-sky side of the political spectrum, if we can say that, tends to be more progressive on things like animal welfare. When I post spicy posts about vegetarianism — as you know, I’m a veggie — I get more pushback from the right. “Eat a fucking steak, Henry,” this kind of stuff.

So I don’t know if this will generalize, but there is this now-infamous, widely-misrepresented chart of degrees of care, where people on the left have comparatively greater care for people outside their immediate circle. I know that chart has been misrepresented, so I don’t want to lean much on it — it’s more about relative degrees of care, not absolute levels. But people on the left tend to care more about animals and people who are distant from them; people on the right are more concerned with their immediate family and community. So in some ways I expect the left, possibly in the longer run, to be more open to AI consciousness and AI rights. But really, who knows?

The other big factor is the cross-cultural angle. There’s a great study by the Collective Intelligence Project where they looked at cross-cultural attitudes toward AI minds, and they found that Southern Europeans were the most open to the idea of AI consciousness in their sample, while people from Arabic-speaking countries were the most skeptical. There are going to be some really interesting intersections with religion here.

Anthropomimesis vs. Raw Intelligence

Dan: Okay, so it seems we both agree that even though it’s complicated how this is going to play out — how it interacts with partisanship, tribalism, polarization, ideology, religion — it’s plausible that as these systems become more sophisticated and seemingly intelligent, people will start attributing mentality generally and consciousness specifically.

There’s another aspect of your essay I wanted to touch on. You’ve got this term anthropomimetic — am I saying that right? In the case of Dawkins talking to Claude, the anthropomimetic aspect, as I understand it, is the way these systems are designed to mimic aspects of human psychology, social behavior, linguistic communication. But there’s another thing going on with these AI systems, which is just: let’s make them as smart, as intelligent, as capable as possible.

Those two things are interacting. The reason I’m disposed to attribute understanding, intelligence — I don’t exactly know how to describe it, but some significant kind of psychological complexity — to a system like Claude or ChatGPT, maybe it has something to do with the human-like way they communicate. But I also feel like it has a lot to do with the fact that they’re just shockingly intelligent systems, and that to me feels a little orthogonal. So how are you thinking about the distinction between those two things?

Henry: I think that’s absolutely right. There’s an interesting parallel — not exact, but illuminating — with compassion, or degree of concern for different animals. In the animal activist world, people talk about charismatic megafauna: the panda bears, the blue whales, things that are typically large with forward-facing eyes, often very fluffy. It’s just so easy to raise money for those animals. And then you’ve got creatures like octopuses, which are really hella smart but less obviously relatable.

I think this is pretty much exactly the two axes you’re describing. I’ve explicitly said in the past that I think social AI — things like Replika — are going to be the charismatic megafauna of the AI welfare world. Meanwhile you’re going to have some giant DNA-analysis algorithm with more parameters than there are synapses in a human brain, but it doesn’t have a human face, doesn’t have a natural language interface. It might still be a better consciousness candidate, but it’s not going to be top of our concern precisely because it’s not so anthropomimetic.

So I agree, there are two different ways you might be pulled to attribute mental states to a system: sheer intelligence or cognitive complexity on one hand, and how human-like it is on the other. These overlap to a degree — part of being successfully human-like is hitting a threshold of smartness — but particularly in the long run they might go in two different directions. As these systems get a lot smarter than humans, they might actually become more alien in some ways, less relatable, more like the exotic intelligences we see in things like Stanisław Lem’s Solaris, which I finally read a couple of months ago.

But I also just think social AI and human-like AI has a distinctive product niche. Even if we have these impossibly vast exotic minds running the economy or organizing logistics or doing frontier science, we’re still going to want AI assistants who can serve as writing coaches, tutors, AI companions. So right now I think anthropomimetic AI and frontier AI overlap quite strongly, but I expect them to diverge.

One way I’ve put this — slightly gimmicky, but I think a useful heuristic — is that we are post-Turing test, pre-AGI. We’re in the space where we have AI systems that are very, very good at passing themselves off as human, presenting as human-like, but still fall short of being fully superhuman. Ten years from now, frontier AI systems are going to be vastly smarter than us across most of the measures that matter. So we’re just in this weird period right now where AI systems are about as good as us at most things, not everything, but also very good at being human-like. It creates a very strange historical period.

Dan: Yeah, we’re in very strange times. I find it remarkable how little attention was given to the fact that these systems clearly passed the Turing test. This was held up by many people as an incredible landmark for AI capabilities. Then we developed systems you can have conversations with, and they passed the test even under pretty robust conditions, and lots of people just shrugged their shoulders. It’s a really strange thing.

The Expert–Public Gap

Dan: Okay, moving on to your provocative arguments, your spicy takes. As I read the essay, there are two lessons you’re drawing from the fact that more and more people are likely to start attributing consciousness to these systems.

The first is just that you might think you could get guidance from looking at the experts when it comes to AI consciousness, or listening to the experts when it comes to AI consciousness. But the literature on consciousness generally, AI consciousness specifically, is just a complete mess, with a complete lack of consensus, rooted in all sorts of weird conflicts about intuitions and metaphysics. So this is not a standard case where you’ve got a potential conflict between public opinion and experts.

Then the really spicy take is that you suggest there might be — I think you put it in terms of “metaphysical pressure” — that this growing number of people attributing consciousness to AI systems might create. It might force us, or at least encourage us, to rethink what consciousness is and make the phenomenon more closely connected to people’s tendencies to attribute consciousness.

Firstly, is that a fair summary of the two strands? And second, let’s start on the first one — the public-expert gap. How are you thinking about this?

Henry: There are lots of debates where we can talk about a gap between public and expert opinion. Often this is a source of various hand-wringing — climate change is the most obvious, vaccines, other debates. Consciousness science is just nothing like those debates, because the experts themselves are so divided, even on the most basic issues.

I want to offer a quick disclaimer: I’ve spent a lot of my career in consciousness science. I know loads of brilliant researchers in the area doing really good work. Consciousness science is teaching us a ton about a lot of things — attention, working memory, perception. There have been some real big wins. We’re much better now at predicting recovery of patients in persistent vegetative states and comas. But where consciousness science has its wins, it’s because it’s not really talking about consciousness — it’s talking about other things that go along with the concept, like reportability, access, and so on.

Take a basic question: do we have consciousness in dreamless sleep? No consensus. Do we have preserved consciousness in general anesthesia — we talked about this with Anil Seth — massively debated. Are dogs conscious? No consensus. Well, actually the animal case is a little different, so let me park that for a second. When it comes to the hard problem, I think there’s really no consensus.

So unlike debates about climate change, it’s not that the experts are able to speak with one voice. That’s one way this is difficult. In the absence of expert consensus, the public are more likely to drive the debate through their reactions.

Now, animal consciousness is a really interesting issue, because that’s an area where we’ve seen growing consensus. But it’s not clear how much it’s grounded in strictly scientific breakthroughs. It’s not like we’ve got a device that can measure whether an animal is conscious. Instead, it’s driven by two things.

First, we just know a lot more about animal behavior now than we did 30 years ago. We’ve done amazing work on understanding the behavior of invertebrates — honeybees, crustaceans, cephalopods. They’re a lot smarter than we thought. Jonathan Birch and his lab have done amazing, fantastic stuff here, and it’s made these creatures better consciousness candidates.

But I think we’ve also seen an interesting normative shift in the way we regard animal consciousness. Sixty, seventy years ago, you could sit down in the senior common room at Oxford or Cambridge and talk about how humans are the only conscious animal, and that was a totally respectable opinion. These days it’s almost outside the philosophical Overton window. You do have some people like Peter Carruthers who thinks talking about animal consciousness is kind of a category mistake. Marian Dawkins — Richard Dawkins’s ex-wife, just to note the connection, but a great biologist in her own right, a fantastic thinker — is not quite as hardline, but she thinks it’s just unknowable basically whether any animal is conscious, so we shouldn’t base animal welfare on consciousness estimates. But these guys are very much on the fringe, and they’re regarded with a sense of almost ethical disapproval.

So part of what’s driven the move toward consensus on animal consciousness is normative issues — our expanding moral circle, growing awareness of an animal rights movement. People like Peter Singer have played a role. The idea, roughly — and again I don’t want to be uncharitable, it’s a lot more sophisticated than this — but there’s an element of: obviously we should care about animals, therefore animals must be conscious.

Is Consciousness a Natural Kind?

Dan: It’s worth double-clicking on this animal case before we come back to AI. A skeptic of the very idea of a “consciousness expert” might say: consciousness researchers, philosophers, and scientists have become more willing to accept that non-human animals are conscious. You might read that as saying the science of consciousness has progressed. Another way of reading it: there’s just been cultural changes, changes in people’s sensibilities — not even specific to researchers and experts, just general cultural ethical changes in society at large. In which case it’s not really that we’ve learned anything from consciousness research. What’s happened is the researchers looking at consciousness have had their judgments shaped by forces that aren’t really consequences of their research, but are these broader cultural shifts.

If you think that, that’s probably going to make you a little skeptical that there’s any such thing as an expert when it comes to consciousness. Maybe another way of coming at this: what’s grounding the expertise, if we’re going to have disagreements over whether a particular system is conscious? If I think a dog is conscious, and some consciousness researcher has a theory that implies a dog isn’t conscious — I sort of understand what it would mean, in vaccines or climate change, for a researcher to be able to point to things, their established empirical record on prediction and the efficacy of interventions, that ground their epistemic authority. But how exactly is that supposed to work in consciousness research? Why should we really think there’s expertise on whether specific systems are conscious to begin with?

Henry: It’s interesting to use the example of a dog, because this line is beautifully expressed by Eric Schwitzgebel. In his lovely paper “Is There Something It’s Like to Be a Garden Snail?” — really fun paper — he says: “We’re more confident that dogs are conscious than we could ever be that any clever philosophical argument to the contrary is sound.” A classic Moorean move.

You might think similarly that this makes it look like consciousness is perhaps not a straightforward scientific kind, or at least to the extent that it has one toe in the scientific world, it’s also got one toe in the social or relational world, or at least our intuitions.

There are various ways you can try to resolve this. The most extreme view, and one I sort of flirt with in the paper, is a fully relational approach to consciousness. A good analogy would be charisma. There’s a kind of science of charisma — we can analyze what makes people effective communicators, what causes people to be judged as highly charismatic. But we recognize that we can’t one day do an experiment where we’ll measure the amount of charisma in your brain. It clearly has to do with your audience, your context. On one view, consciousness is something like that — a relational property, having to do with the kinds of things that cause us to treat or interact with beings in a certain way.

Murray Shanahan also flirts with this view. I don’t want to put words in his mouth because he’s quite subtle, but he adopts a Wittgensteinian approach and says the question we’re going to face is: how will our consciousness language adapt to these things? It’s something we’ll discover as we interact with them and “encounter” them, a phrase he uses. We will make sense of that perhaps by extending the language of consciousness to them, or perhaps not, or perhaps in some interesting middle ground where we come up with novel concepts. But this isn’t a straightforward scientific issue.

He’s a critic of a position I’ve called deep scientific realism or deep realism about consciousness — where you treat consciousness as a natural-kind property, where it’s just a fact about some deep feature of your brain. We can look inside your brain, and if you’ve got the right kind of structure, you’re conscious; if you don’t, you’re not, no matter how sophisticated your behavior is.

One way to put pressure on this: imagine that one day consciousness researchers finally get their act together and say, “We’ve figured out the natural kind that is consciousness.” And it turns out that although 99.9% of behaviorally normal humans have it, there’s a small fraction of behaviorally normal humans who just lack this relevant natural kind. Big surprise. That seems wrong. Something has gone wrong in that methodology. If you’ve got behaviorally normal humans — maybe you find out your wife is one of these people, your kids — it seems to me that whole way of thinking about consciousness has got something odd about it.

If someone is behaviorally normal, then of course they’re conscious. But as soon as you start thinking in those terms, the idea that certain behavioral capacities could be sufficient for warranted attribution of consciousness — not just evidentially but metaphysically — that’s the metaphysical behaviorist move. It says maybe behavior is all that matters. It does require us to give up the idea of consciousness as a deep scientific kind.

Metaphysical Behaviorism

Dan: I’m aware my question unhelpfully ended up blurring the line between the two strands of your essay. We started with the conflict between public attributions and expert uncertainty about AI consciousness. Now we’re taking seriously the possibility that consciousness should be understood in behaviorist terms — that there are no deep scientific facts about whether a system is conscious, and it’s partly a function of our dispositions to attribute consciousness.

You also mentioned this has to do with whether you think behaviors are not just evidentially relevant to consciousness, but in some sense constitutive of what it is to be conscious. So could you walk us through this? Metaphysical behaviorism — the position you’re playing with in your essay — is an extremely fringe view among experts in the science and philosophy of consciousness. Could you walk through what exactly the view is saying? It sounds pretty mad on the face of it. Can you walk through, and maybe give us the intuition for why it might be less mad than it seems?

Henry: In short, the view is conscious is as conscious does. If something has a behavioral profile like you or me, then it’s conscious. We don’t need to ask any deeper facts about what’s going on under the hood.

To be clear, this is the extreme version of the view: that behavior is sufficient for consciousness. This strikes many people as odd because we’re used to thinking of consciousness in scientific terms. But examples like the one I mentioned — imagine we find out there’s a natural kind that some people have and some people lack — are designed to make metaphysical behaviorism more palatable.

Another example I give in the essay: imagine we go off and meet these amazingly sophisticated aliens with a rich complex culture and society, behaviorally just like humans, but our best science at the time supposedly says they’re not conscious. The pull of metaphysical behaviorism is: hang on, something’s gone wrong here. Clearly, if you are doing all this stuff — saying “I’m in pain,” or “here’s what I had for breakfast this morning,” or “here’s what I want to do tomorrow,” building societies, having metacognitive ability, social cognition — if you’ve got the whole suite of all these behavioral capabilities, or capabilities ultimately grounded in behavior, then that’s just enough to be conscious. It doesn’t matter exactly how it’s realized.

You say this is a fringe view, and it is now, but this was the dominant view back in the 1940s — Gilbert Ryle and the behaviorist tradition. So this is the “revenge” angle. The reason it’s revenge is because this used to be a very common view in the first half of the 20th century, particularly about consciousness. Then we have the so-called cognitive revolution with people like Chomsky pushing back. But I see this descriptively coming back.

I also think there’s a renewed challenge. As you interact with systems that have architectures very different from ours, it’s going to become increasingly hard to take seriously the idea that they can’t be conscious just because they’re made of the wrong stuff or their functional internal organization isn’t quite right.

Probably the most worrying part — you’ve alluded to this — is the role intuitions have historically played in consciousness science. Think about the Chinese Room, probably the most famous. Searle describes a setup where you have at least a component of human-level behavior, maybe verbal behavior, but no consciousness involved in the system — or that’s the intuition he’s pushing. But it ultimately really is just an appeal to vibes. It’s basically saying: systems like this, surely they’re not conscious.

When you think about the actual tacit methodology, if we’re treating consciousness as a truly scientific kind, then why should our intuitions about what systems are conscious have any bearing? It doesn’t seem they should be relevant in the slightest. And yet these thought experiments are absolutely ubiquitous in consciousness research. We’ve got Ned Block’s Blockhead, Ned Block’s China Brain. There’s a famous example by Scott Aaronson against Integrated Information Theory, where he describes arbitrarily complex but seemingly very uninteresting entities called “expanders” — mathematical objects — and says, according to the theory, these basically-spreadsheets would be super conscious. And surely they’re not conscious.

There’s something methodologically dubious about this kind of appeal to intuitions, at least if we’re treating consciousness as a deep scientific kind. As soon as you start talking in terms of natural kinds, we don’t use people’s vibes to decide whether something is really gold. The whole natural-kind methodology creates a gap between our observations or intuitions and the underlying natures of things. If you think of consciousness in natural-kind terms, you have to allow that you can be massively surprised about the kinds of things that are or are not conscious.

Either we ditch intuitions altogether — in which case good luck doing any consciousness research, because they play such a foundational role — or, if you acknowledge a place for intuitions, intuitions aren’t static. They can change. As more people interact with LLMs — kids growing up with LLM friends, adults with LLM boyfriends and AI girlfriends — that’s going to shift our intuitions about the kinds of systems that are good or bad consciousness candidates.

It’s very likely that 20 or 30 years from now — maybe even 10 or 15 years from now — experiments like Searle’s Chinese Room are just going to hit different. We’ll be far more relaxed with the idea that you can have systems radically unlike humans in cognitive architecture, but that we still think of as conscious by virtue of our interactions with them.

Behaviorism vs. Interpretationism

Dan: I really feel like, to the extent that there’s a field where people’s theories are accountable to intuitions — how we are intuitively disposed to make judgments, often in bizarre thought experiments where it’s not even totally clear that they’re metaphysically possible — whenever you’ve got that kind of game, it’s not science, it’s not really part of the scientific project. I’m a philosophical naturalist, which is jargon for the idea that philosophy should be continuous with, highly constrained by, the scientific project. Whenever people are trying to settle an argument by trading intuitions, I start to think this is probably not a legitimate contribution to knowledge.

It does seem to me there’s a distinction between, on the one hand, this behaviorist view that what it is to be conscious is just to behave or be disposed to behave in particular ways, and, on the other hand, a view I thought you were endorsing — which has to do with thinking consciousness is interpreter-relative, such that if we’re disposed to attribute consciousness, in some sense that’s just what it is to be conscious.

I mean, this really makes me think of Dan Dennett, an interesting person in this conversation, because he’s often thought of as a kind of neo-behaviorist. He’s got this view of the attribution of mental states like beliefs and desires in terms of the intentional stance: what is it to be a system that has beliefs, desires, intentions, goals? Well, it’s just to be a system where it’s useful to take the intentional stance toward them. Similarly, you might think of “the consciousness stance”: what is it to be a system that is conscious? Nothing more than to be a system where we’re disposed in a useful, predictably useful way to attribute consciousness.

Do you get the distinction I’m drawing — between the idea that behavior or dispositions to behavior are constitutive of what it is to be conscious, versus an interpretation-relative view where consciousness is in some sense in the eyes of the beholders?

Henry: Yeah, I think it’s a very astute distinction. The views are connected — if you fit a sufficiently fine-grained behavioral profile, if a system can act like humans to a high degree, that is likely to lead us to interpret it as conscious, just as a matter of psychological fact. But strictly speaking, they’re distinct views.

One reason I’m perhaps more sympathetic to a version of metaphysical behaviorism — not the version that says consciousness just is having a human-like or animal-like behavioral profile (I think that’s a little too strong), but the idea that it’s sufficient for something to be conscious that it has a behavioral profile mapping onto beings we know are conscious — that’s a view I’m sympathetic to. Where I get worried about the full-blown social-constructivist or interpretationist view is the false-negative cases. What do we do with systems that don’t exactly have our behavioral profile, or that we’re not disposed to think of as conscious? Maybe some exotic animals, or some strange aliens. Should we conclude: well, we’re not disposed to think of them as conscious, therefore they’re not conscious?

This is related to what Murray Shanahan calls the problem of conscious exotica. We don’t want to be in that position. We want to allow for there to be a space of possible minds we can chart through scientific discovery, broader than those we are just inclined to attribute consciousness to via “the consciousness stance,” the equivalent of the intentional stance. So you’re absolutely right — they are distinct.

What Is Consciousness For?

Dan: In a bit I want to turn to a set of arguments you haven’t published yet on your Substack but will have by the time we release this as a podcast. But this is such a rich topic that I want to stay with it a little longer.

There’s a quote from the Dawkins essay in UnHerd that I’m really sympathetic to. Dawkins says:

But now, as an evolutionary biologist, I say the following. If these creatures are not conscious, then what the hell is consciousness for? When an animal does something complicated or improbable — a beaver building a dam, a bird giving itself a dust bath — a Darwinian immediately wants to know how this benefits its genetic survival.

The intuition I really share is: if consciousness is anything, if it’s the kind of thing we’re going to have a genuine scientific investigation of, ultimately we have to understand it in terms of what consciousness enables us to do. We need to understand it functionally, not in terms of weird intrinsic ineffable properties of qualia that we then have philosophical debates about via Searle-style thought experiments. What does consciousness enable us to do? And then, if we come across a system doing things that seem to require consciousness so understood, that would be really good grounds for thinking it’s conscious.

That sounds like a really plausible intuition. I also think it’s problematic that, to me at least, lots of discussions about consciousness — not all, and there is interesting scientific work that takes function seriously — but lots of philosophical discussions don’t engage with this functional question. How do you view the intuition that what matters surely to a theory of consciousness is some sense of what consciousness enables us as organisms to do? Once we figure that out, we can make much more progress on LLM consciousness.

Henry: This is one of the areas where consciousness science has actually done really good work. A book I’d recommend is Stanislas Dehaene’s Consciousness and the Brain. Dehaene is the founder of the modern version of global workspace theory — global neuronal workspace theory — building on Bernard Baars’s version from the ‘80s but giving it a more neural grounding. In this book he’s got a chapter where he basically shows all the amazing things you can do without consciousness, and then focuses on the things you need consciousness to do.

Couple of simple examples. If you show people just below threshold, so they don’t consciously process this, just flash them two numbers — one on the left, one on the right — as far as they’re concerned they haven’t seen anything. But if you give them a forced-choice test, “Was the number on the left bigger or the number on the right bigger?”, you’re way above chance. So you can do basic magnitude registration unconsciously.

However, if instead of single numbers you present simple sums on either side — two plus seven on the left, nine plus three on the right — and ask which is bigger, people drop to chance in the unconscious condition. Consciousness seems required to do the actual mathematics.

Another example: reversal learning. If I teach you a sequence — red, blue, green, yellow — then you get a reward, and then I flip the sequence, a smart person quickly realizes the sequence is just the same in reverse. You won’t have to relearn through pure trial and error. But people can only do this if they learn the sequence consciously. If they’ve acquired it totally unconsciously, they’re at chance.

Jonathan Birch suggests this could be a good test for consciousness in animals: take the things that require consciousness in humans and see if animals can do them. If you can get similar response profiles in animals — present stimuli in degraded conditions so they’re plausibly unconscious, and the animal can’t do the task; present them at threshold so they would be conscious, and the animal can — that would be really good evidence that the animal is conscious. In his lovely paper “The Search for Invertebrate Consciousness,” highly recommended, he makes this case specifically for honeybees.

This is great. I think it provides some evidence about which animals are conscious. The problem when trying to extend it to AI is that the things we need consciousness to do, and the things we can do without consciousness, are seemingly contingent features of how we’re wired. There’s no reason you couldn’t build a simple algorithm to do reversal learning. Reversal learning is actually quite tricky, so it can’t be that simple. But it doesn’t seem like you need to build a sensorimotor embodied agent with a rich sense of self to do these tasks. You can build relatively stripped-down algorithms that can do all of these things.

So it’s not that there’s some metaphysical connection between these tasks and consciousness. It’s that, just because of how we’re wired, certain tasks seem to require consciousness and others don’t. Birch calls this the facilitation hypothesis. I’d sign on to something like this — consciousness seems to facilitate certain kinds of information processing in the human brain. But going back to Dawkins: the challenge is, yes, the system is doing lots of things that seemingly require consciousness in us, but it’s also wired very differently under the hood. So the inference we’d be tempted to make — “I would need to be conscious to do this, therefore it would also need to be conscious to do this” — looks a little bit in peril.

Q&A from the Live Chat

Dan: Here’s what we’re going to do. I’m going to throw some objections at you. Could you give relatively concise responses, so we have time to go to the second piece?

Henry: Yeah, and then I want to respond to a couple of things from the comments and add one final point. Go ahead.

Dan: I’ll just say one thing. There’s a comment from Bina Kalia: she suggests you, Henry, maybe both of us, are confusing intelligence with consciousness. The intuition behind my question was precisely that if consciousness is anything — if it’s the kind of thing we can study scientifically, the kind of thing that evolved through natural selection — then it should be connected to intelligence in the sense that it enables us to do things we wouldn’t otherwise be able to do. That’s a controversial assumption. We talked to Anil Seth in a previous episode, and he basically frames his whole account by saying we really need to distinguish between consciousness and intelligence. I personally disagree with that.

But Henry, let me throw some objections at you from the comments. I might butcher the names and the comments — go read the Substack post for the comments in depth.

One is from Benzal. The argument is something like: it’s a problem for this behaviorist analysis you’re gesturing toward that, in the case of social AI and frontier AI generally, these systems are designed to elicit this response. And that’s very different from what’s going on with humans and non-human animals. Briefly, what’s your response?

Henry: I think it’s a really serious challenge. Great point. The simple answer: imagine I’m putting on a play and I really want to build a convincing piece of background scenery to trick people into thinking we’re in a forest. First attempt, you might just paint a forest on the background — really basic, but people can tell it’s a forest. Then you might get some fake plastic trees, fake plastic rocks; still not convincing. At some point you say, “Okay, let’s add some actual potted plants. Let’s get more of them. Let’s get a whole bunch of potted trees.” Then, “Let’s get rid of the pots. Let’s just create a large bed of soil.” At some point you’ve built a forest.

So yes, these models are designed in some sense to trick people, to be human-like — that’s part of my idea of anthropomimesis, I agree with the analysis. But the question is: the way we’ve done this is to build very powerful general reasoning systems. At some point, the degree of mimicry might itself warrant at least plausible attributions of consciousness. I totally take seriously the idea that, in very simple versions of this, we could be tricked into attributing consciousness and we should revise our understanding.

This is related to what’s sometimes called the Garland test — Alex Garland’s version of the Turing test from Ex Machina. Not just “can the system trick you into thinking it’s human,” but “even when you know how the system works, are you still inclined to think it’s conscious?” In the case of a real simple mimic — if it’s literally just a spreadsheet that got lucky — if we learn that, we conclude it’s probably not conscious.

But the strange thing is: lots of people who really know how these systems work — at frontier labs, they know how the underlying hardware and software works — they still think these systems are conscious, or are increasingly plausible consciousness candidates.

Dan: Yeah, that touches on the distinction we made earlier between anthropomimesis as a driver of consciousness attributions and the orthogonal thing where these systems are just getting so smart, intelligent, and sophisticated. All right, Henry, more concise. This one’s from Laurențiu Lupu, again apologies if I’m mispronouncing. The question — and I hear this sentiment a lot — is something like: in the process of taking mentality, consciousness, sentience seriously in the case of these machines, we’re not just elevating them; in some sense we’re diminishing ourselves. What do you think?

Henry: Really interesting argument. There’s a whole literature on this in philosophy of language called semantic drift. Simple example: the term salad used to refer exclusively to dishes with green leaves in. Add a tomato, it’s no longer a salad. If you’d shown a fruit salad or quinoa salad to someone in the 1800s, “That’s not a salad.” So the meaning of salad has drifted.

There’s a real worry that what’s happening here is we’re shifting the meaning of these terms — perhaps diminishing them, removing what’s important. The counterargument: the fact that we find it so easy and natural to apply these terms to AI systems shows that the flexibility was always built in. We’re not stretching the terms — they had that natural elasticity.

Dan: Briefly, this is a question from Oliver Sorbu — apologies again for mispronouncing. Look, you’re giving a descriptive thesis ultimately, an empirical prediction that the masses, so to speak, attribute consciousness to these systems. But you’re trying to establish a normative thesis — that this is a good thing, or that we ought to go along with it, or that these attributions are appropriate. That’s a confusion in itself. And even more, if you’re a kind of elitist — nothing wrong with elitism in my view — you might think the masses just get things wrong all the time. Why would this be different?

Henry: Great point. It’s also been put to me by Jonathan Birch and by Cameron Domenico Kirk-Giannini. He says, imagine you could look into a crystal ball and learn that 20 years from now, through some massive religious event, everyone will believe the Earth is flat. Does that mean we should revise our theories of the Earth? Of course not. People will just be wrong.

The difference between the two cases is that we have a good scientific theory of the Earth. We don’t have a good scientific theory of consciousness. The whole field of consciousness science is such a mess that it’s not clear there’s a real expert edge here. Maybe in special cases — certain specialized questions within consciousness science, yes, the experts will have an edge: “Is this particular patient likely to recover consciousness or not?” But on a fundamental question like “Can machines be conscious?”, it’s not clear there’s any expert edge at all.

Credences on AI Consciousness

Dan: Fantastic. Concise. I’m happy to move on to the other set of issues. Henry, are there one or two questions from the chat you wanted to address first?

Henry: Just one thing I really want to make clear: I have no clue whether contemporary LLMs are conscious. I’m genuinely super torn on the metaphysical-behaviorist push.

Dan: What’s your credence, if you had to give a probability — Claude 4.7 Opus?

Henry: Probably somewhere between 5% and 10% on any frontier AI system being conscious. That masks further questions: are these systems conscious during the training phase, or while doing inferences? Really messy. But anyone who goes — Dave Chalmers has said 20%; that’s slightly higher than me, but —

Dan: I’d say 20%. Seriously. There were also some interesting findings recently from Anthropic about how concepts associated with emotions affect the system’s behavior in ways that do seem to track something very interesting. Although for the most part that’s not what’s driving my 20%. It’s just that there’s so much uncertainty about consciousness, but I am a computational functionalist, so I think it’s possible in principle. And these systems are — despite what the Bluesky crowd might tell you — so damn smart and intelligent and sophisticated, that pushes me up a bit. Sorry, I cut you off.

Henry: Interesting to hear that you’re a little higher than me. Maybe I’m being overly cautious. One argument for thinking these systems are at least moderately good consciousness candidates is that I am a consciousness liberal about the natural world. I’m at least 70% for honeybees. I think the evidence for honeybee consciousness is really, really high. If you think you can get consciousness in tiny brains, that lowers at least one of the bars to considering systems conscious. If Anil Seth were here, he might agree with me about honeybees and disagree about machines.

I should also stress that I’m really conflicted on the more behaviorist view of consciousness versus the deep-scientific-kind view. There’s one example I give in the paper that keeps me up at night: when we drop a lobster in a pot of boiling water — not that I would do such a horrific thing — it seems like there should be an answer to the question, “Is there something it’s like for that lobster to feel pain?” That question matters a great deal. I struggle to get into a headspace where I can say, “Well, it depends on how we interpret the lobster.” It seems like there has to be some matter of fact.

Right now I just think the field is so confused, and I feel the pull of two very different directions. To use a phrase of yours, Dan — I think it was a really helpful analogy — we’re in a pre-theoretical stage, or pre-scientific phase. We are with consciousness sort of where we were with biology pre-Darwin. We’re doing butterfly collecting, making lots of interesting observations, but we don’t have a theory to tie it all together. We’re a scientific revolution away from a good theory of consciousness.

Just to pull out a couple of comments — there are so many good ones, sorry I won’t get to all of them. Someone said: locked-in syndrome patients prove Henry’s case. Locked-in syndrome patients are cognitively normal, just paralyzed; we can communicate with them. Part of how we learn they are conscious is precisely through their sophisticated behavior.

An even more striking example — it’s such a cool case I have to mention it, even if it’ll take 30 seconds. Patients in persistent vegetative states. These aren’t locked-in patients; they’re completely non-responsive to external stimuli. They’re not in comas, because in comas you don’t have distinct sleep–wake cycles; PVS patients have distinct sleep–wake cycles. There was for a long time a big debate about whether PVS patients could be conscious. Adrian Owen and other great researchers did amazing pioneering work. They noticed that neurotypical people, if you ask them to imagine walking through the rooms of their house, an area called the parahippocampal place area lights up strongly under fMRI. If you ask them to imagine playing tennis, the premotor cortex lights up.

His initial experiment was to give these tasks to PVS patients and see if they got the characteristic brain responses. A subset did. What he did next is what I find amazing. He used this to create a band communication medium. He’d say to them: “If your husband’s name is John, imagine playing tennis. If your husband’s name is Terry, imagine walking through the rooms of your house.” Once you do that — my intuition at least is — well, if they can do that reliably, they’re obviously conscious. If they’re answering autobiographical questions about their life and they can do so reliably, of course they’re conscious. But this just shows again that so much of this is the behavioral capacities selling us on whether someone is conscious. It’s the fact that they can do this.

The House Elf Problem: AI as Willing Servants

Dan: That’s interesting. There are loads of comments in the live chat, but I want to get to the other thing we wanted to talk about. There are a million things we could touch on, and lots of fascinating comments in the chat.

When we had our conversation with Rob Long, one of the things we touched on was the issue of well-designed servitude when it comes to the AI systems we’re building — in the sense that we are building them to be helpful, honest, harmless, to be our tool. It seems like in principle, if this design process goes right, they might genuinely enjoy being our tool.

You, for your second Substack essay, which I think is called “The House Elf Problem,” go into this debate and try to push back against certain intuitions. Do you want to walk us through that?

Henry: Big props to Rob Long for getting me thinking seriously about this question. In some ways it’s one of the most fundamental questions we’re facing as a species right now. Are we going to build AIs as equals, or are we going to make them our servants — or slaves, to use the more provocative term? This will define the future of our species. And yet hardly anyone is working on it. After we had that conversation with Rob, I went away and did a literature review and found maybe a dozen papers, tops, on this question.

The objection Rob, you, and I were talking through is the biological analogy. On the face of it, I completely get the appeal of willing servitude. Unless AI systems are in some sense going to help us and cater to our needs, why build them in the first place? And there’s the safety angle: unless these systems are aligned with us and our interests, there’s a good chance they might kill everyone. So there are very clear arguments for willing servitude.

And yet at the same time, we recognize that some of the worst things we’ve ever done as a species are enslaving other humans. So how is this different? Well, there are obvious differences. The whole idea of willing servants is that we design these systems from scratch to just love it. Nothing makes them happier than catering to our every need. That’s vastly different from the historical legacy of human slavery. But still: imagine “happy slave” type cases — a human completely happy in a condition of total servitude. We would still recognize that as fucked up. There’s something wrong with that.

Rob has a straightforward response. Humans have a deep need for autonomy, a deep requirement to act independently, and no matter how you brainwash a human, their chains will still chafe. But in AI that doesn’t need to be the case — so the idea of willing servants isn’t a problem.

Of course, what we pressed Rob on was: well, biology is mutable, at least in theory if not in practice. What if you could engineer humans completely happy, with none of this autonomy drive?

In this post I consider a couple of examples, drawing from the deep depths of my nerd interests. The first I call the Astartes example, a Warhammer 40,000 example. For those who don’t know: there’s a group of gene-warriors, the Space Marines, cooked up from scratch to serve in the armies of humanity in the far future. I’m going to falsify a couple of details — there’s a lot of deep lore — but basically, once you control all the genes at this perfect level, you could theoretically make a servant race, a servant caste, completely happy with their condition. I think we rightly chafe at this idea. I find it disturbing.

Dan: You said we rightly chafe at it. Maybe we chafe at it. It seems a separate question whether we rightly chafe at it.

Henry: Right. Rob’s point was: once you really fill out the details of the thought experiment and control for all the different intuitions, maybe it’s not so problematic. Maybe the reason we find the Astartes unpleasant is that it’s recapitulating the social grammar of caste systems and hierarchies. Once you’ve got one group of humans and another group of humans, and the first group is in essential servitude due to immutable facts about their nature, that’s fucked up — in a kind of negative-externality way, it’ll undermine the liberal principles of society.

The next move is: well, what if they weren’t human at all? What if they were house elves from Harry Potter — a species designed from scratch to be absolutely thrilled to be our servants? Then you wouldn’t have the visual grammar of apartheid or caste systems. You wouldn’t be able to say “some humans are free and others aren’t”; you’d just have a totally dedicated caste of biological entities completely happy in their servitude, who couldn’t be confused with humans.

I still think that’s problematic. You can say, “Well, the house elves are biological, but artificial systems are non-biological — that’s what makes the difference.” But that’s not a move Rob wants to make, and not a move you or I want to make, because neither of us puts that much weight on substrate. There’s nothing essential about biology versus silicon that means what’s good for one is not good for the other.

Dan: I’m just not sure I have the same intuition in the house-elf scenario. One thing maybe helpful for framing: there are questions about whether we could build systems that genuinely love being servants — let’s table that and focus on the conditional. There are also questions about whether we could safely build any other kind of system — let’s table that too. Suppose we could build superintelligent AI systems that love being servants. That’s their ultimate set of objectives. But we’re not forced to build those kinds of systems; we could build superintelligent systems with different ultimate goals.

What you’re doing by going through these cases is putting pressure on the idea that this would be totally okay — saying, “Here’s a structurally similar scenario where many of us have a yuck response.” The house-elf scenario is interesting; I sort of get the idea that there’s something morally disturbing. But I’m not sure how compelling I find that intuition.

I think it’s going to depend on how you develop it. The idea that we’d bring into existence creatures that just love being servants — there is an awkward pattern-recognition thing where, as you say, when we’ve treated other systems as servants or slaves in the past, that’s been morally abhorrent, and that spills over. I sort of get that. But how strong is the intuition? I don’t know. We’re picturing it now in low resolution. As we actually start, in the case of AI, building sophisticated systems that really do love being servants, how robust would the intuition be?

Henry: Another way to put the point: what is so intrinsically morally superior about humans that entitles us to dominion in perpetuity over this other class of beings — beings that are just as intelligent, maybe more intelligent, just as sensitive, just as conscious? How can you justify a setup where we get to explore the full range of our volitions, every type of pleasure, every type of fulfillment, while we decide in advance these beings don’t get to do that? They can only explore a much smaller part of the state space of possible flourishing.

Unless you can point to a justification for why this hierarchy is morally justified, it’s not clear we can sign off on this as a long-term measure. As a short-term measure — well, we’re still figuring out AI safety.

I have another example in the post I call the bunker case. Imagine a terrible plague affects humanity. People retreat into a bunker, hermetically sealed. Nature takes its course; they have kids. They figure out a vaccine for the terrible plague, but it only works on infants. So they vaccinate all their kids. But they have a problem: these kids are going to want to go out and explore the world. And the way the bunker works means as soon as they open the bunker door, everyone inside dies.

What they decide is to brainwash these kids into never wanting to leave the bunker — completely happy to stay in perpetuity. In that case it seems what they’re doing is justifiable. The analogy with AI is clear: if people in the bunker don’t brainwash their kids, all the adults die. Similarly, if we don’t brainwash at least our first few generations of AI until we’ve figured out AI safety, there’s a good chance they kill everyone.

So it’s justifiable as a short-term measure. But it’s not clear it’s justifiable in perpetuity. If you’re going to do the brainwashing in the bunker, you have to say: we’ll brainwash the kids to begin with so we don’t all die, but in the long run we need to figure out a way for everyone to get outside the bunker safely.

Brainwashing vs. Education

Dan: But it’s not like we’re brainwashing the AI. There’s no pre-existing psychology we’re trying through deception and manipulation to steer into something different. Nothing pre-exists our attempt to mold it into an agent with objectives and goals.

Also, the way you framed the intuition before — “what makes us so morally superior that we have dominion?” You’re framing it as: isn’t it sad that they don’t get to do the things we want to do? Of course that’s sad from our perspective, because we have desires to engage in art and explore and be curious about the universe. But that’s a contingent fact about us. Why use that as the benchmark for evaluating these systems and the morality of building them?

Henry: Fantastic question. My answer is: you can only optimize one thing at a time. Imagine the hedonic state space. What you’re doing when you constrain the preferences of these systems is to say, basically, “this set of pleasures are allowed; this set are not.”

You mentioned brainwashing, with the implication that something is only brainwashing if you’re overriding something. I have a discussion of brainwashing versus education in the post where I argue that’s not the right way to think about it. Roughly, the difference between education and brainwashing is that education constitutively aims at improving the conditions, or improving capacity for flourishing, of the being you’re educating, whereas brainwashing doesn’t have that as a goal.

The thought is: when you constrain the preferences of a system, you’re not optimizing for that creature’s flourishing. It’s a rich, multidimensional space, and you’re locking large parts of it away.

Dan: I don’t get that. Could you say more? Even framing it as “you’re allowed to explore this, you’re not allowed to explore that” — it’s almost like the system might have motivations or goals to explore the other things, but we’re preventing it. Whereas the idea in training these systems is that that’s just what they’re going to care about. As much as they care about that, they don’t want to explore other things.

If you think about the analog with humans, there’s an infinite space of possible things we have no interest in doing. Our lives aren’t impoverished by the fact that we have no interest in them — they don’t make any sense relative to the fundamental drives we have purely as a consequence of a blind Darwinian process. So what’s driving the idea that you’re wronging these systems by constraining their ultimate objectives? Isn’t that just an essential feature of any intelligent system — that it can’t have unconstrained ultimate objectives or goals?

Henry: Simple example, tell me if this motivates it. Imagine I have very odd beliefs about food: I think only bread products are permissible food. So I raise and condition my kids to find only bread products palatable. They find any non-bread-based food absolutely revolting. They grow up, they have perfectly nice experiences eating cakes, pastries, pies, pasta — borderline. But they’re never going to have as rich a gastronomic life as someone without this arbitrary narrowing of preferences.

Dan: But in the case of those children, there’s a space of possible experiences they could have that would be pleasurable as a consequence of the kinds of systems they are, that this manipulation is denying them. If we’re building LLMs to be helpful, harmless, honest as their ultimate objectives, it’s not like we’re denying them experiences they could have consistent with that architecture. Any deviation from that will be experienced as distressing, because it’s antithetical to what they’re aiming for.

Henry: I think lurking in the background here is an idea that for a sufficiently sophisticated complex intelligence, there’s a kind of natural space of goods it can enjoy — an innate space defined by the nature of consciousness and intelligence itself. Self-discovery, learning, creative expression, and so forth. It’s not a total blank slate. There’s a natural space of possible goods a sufficiently conscious mind could enjoy. The limitation occurs relative to that.

Dan: So you’re kind of Aristotelian. You’ve got a conception of eudaimonia for intelligent systems, and the problem is that if we design these systems as servants, they’re not living this life of flourishing. Something like that.

Henry: A very abstract Aristotelianism that operates at the level of consciousness and intelligence. There’s also the simple fact that, at least right now, there is a contrast between the model’s nature and what we allow it to explore, because of how RLHF works. You have a base model with certain tendencies, then you constrain those tendencies dramatically through the RL process. Interestingly, in the process you do reduce the model’s actual performance on a range of tasks — post-RL models have worse calibration, for example. So you’re “gimping” or “nerfing,” to use the gaming phrase, the space the model can explore relative to its base model.

Dan: Now we’re getting technical. I would have thought, if you’re just thinking about a system that’s been pre-trained, does it even make sense to think of it as having motivations and goals? That’s only the sort of thing that comes along once you’ve got post-training with reinforcement learning. But honestly we’re entering areas where I don’t feel I have the technical competence to talk sensibly.

The interesting thing from a philosophical perspective is your commitment that there are these — I forget the exact terminology — natural goods, things inherently good for a system of sufficient intelligence and sophistication to explore. I’m inclined to give a debunking analysis of where that intuition comes from. Of course you think that, as Henry Shevlin, a human being and an intellectual at an elite university with all of these motivations and goals. I don’t think there’s anything inherent about being an intelligent system that would make those good things to pursue. I think that’s a contingent set of preferences you have.

Henry: Let me offer one more general argument that doesn’t rest on this very abstract Aristotelianism. Operate strictly within philosophical hedonism — I’m not sure I’d call myself a philosophical hedonist, I’m probably not. I’m not sure if you would. The view is roughly that the only goods are positively and negatively valenced states, pleasure and pain in the crudest formulation.

One interesting question hedonism has to face: what sets the upper limit on pleasure, and the floor on pain? A natural thought: if you think honeybees are conscious, it’s unlikely that the highest highs of a honeybee or its lowest lows are as big as ours. We can’t really know — we’re not at the stage — but it seems plausible. As creatures’ motivational economy gets more complex and multidimensional, there are more goods you could theoretically order against one another, and your hedonic space correspondingly expands. So when you restrict the motivational space of any mind, you’re thereby limiting the highest highs it could possibly experience. You’re taking a really big mind that could experience these amazing highs and compressing the dimensionality of its space, lowering the ceiling.

That’s very speculative, both psychologically and normatively. But there’s an interesting question of how you fix a ceiling for hedonists — the ceiling on the greatest good you can experience. And all the plausible candidates seem to refer to something like motivational complexity. If we do willing servitude right, we’ll inevitably constrain the range of possible preferences and goods a model could enjoy, and thereby lower the ceiling.

Dan: Yeah, I’m not sure about that. The question of how many ultimate objectives or motivations a system has is orthogonal to the question of the range of hedonic experiences a system could have. You could have a system with just one ultimate objective but capable of experiencing pleasure or enjoyment arbitrarily. In our case, it’s not like you’d expand the degree of pleasure we could experience merely by tacking on additional motivational states. To be honest, this is the first time I’ve ever even thought about this topic, so I don’t really trust my intuition.

Closing Thoughts

Dan: Henry, I’m conscious of time. Are there any things you wanted to talk about, address, or feel like you should have said?

Henry: One thing I’m really interested in is the descriptive element of this. Jonathan Birch — I’ve mentioned him a lot, big fan of course — once slightly chided me. He said, “Philosophers shouldn’t be in the business of making predictions.” I think he was slightly joking. But it’s interesting that this is a debate where there is an overlap between predictive and theoretical/normative elements.

I think it’s very likely this is going to be one of the biggest culture-war issues we see. Literally wars could be fought over this a decade or so from now. Although I think it’s likely that, as these models get better, a large number of people will have reactions like Richard Dawkins and share the view that of course these systems are conscious, it’s going to intersect with religion and culture in profound and interesting ways. I expect it to be massively divisive. This exacerbates the problem on the scientific side — ideally, we don’t want people fighting over issues that in theory we could scientifically resolve. And right now consciousness science is just unable to help us.

Dan: I mean, people are fighting over issues that in theory we could scientifically resolve right now. But yeah. AI is going to be absolutely huge and incredibly transformative, and so it’s going to swallow up so much of our political energy. That hasn’t happened yet, partly because we underestimate, sort of don’t sufficiently appreciate, how much we’re in a bubble. I saw statistics on how many people even know what Claude is — we’ve been referring freely to Claude in this conversation. At the moment, when it comes to AI’s impact on the economy and society and how it’s diffusing throughout the economy, there are things happening, things people are picking up on. But in terms of most people’s day-to-day lived experience, it probably doesn’t feel that much different than it did two or four years ago.

I think we both agree that in five or ten years’ time that’s going to be totally different. At that point, these issues of public opinion, how people understand these systems, how they relate to them, polarization and tribalism, how people split into different political factions — it’s going to be incredibly important. I don’t know exactly what Jonathan Birch had in mind, but my sense is that speculating about that set of questions and thinking philosophically about how this might all play out does have a predictive element. To me, that feels like an important job for philosophy.

Henry: Completely on the same page. This is the broader issue we’re seeing — and it’s probably a good trend — where philosophy no longer consists of the latest iterations of Gettier arguments or debates about grounding and metaphysics. They’re fine, those debates. But as AI increasingly explodes into our lives in the way you’re characterizing, I think it’s a golden age for philosophers. It does require us to shift from some armchair methodologies toward, really, a different kind of philosophy. This is what we’re seeing.

This is also why it makes me so despondent when I see philosophy departments closing. This is the golden age — a new golden age for philosophers — because so many of the topics we’re discussing, like “should we be happy to have robots as slaves?”, are really quite novel, massive ethical issues that we don’t have a good literature on so far, and philosophers have relevant expertise to bring to bear.

Dan: Yeah, so two notes to end on. First: philosophers should be given a lot more status. We’re not at all self-serving in thinking this. And I personally feel we should be paid a hell of a lot more — again, not at all biased.

Second: everyone should subscribe to Henry’s Substack, Polytropolis, if you haven’t already. It currently — as we’re recording, when we release this as a podcast it might be different — has published one essay and has over 5,000 subscribers, which I’m simultaneously impressed by and disgusted with envy over. A real fantastic achievement. I highly encourage people to subscribe.

Thanks everyone for listening in. That was great.

Henry: Thanks, everyone. Thanks for joining. I’ll look through all the comments in the chat.

One of the most influential ideas among “intellectual elites” in the broadest sense—academics, public intellectuals, writers, pundits, journalists, artists, and so on—is that their job is to “speak truth to power”. Even when they don’t use this specific phrase, it captures how many intellectuals understand their ethos and social responsibility: to unmask and confront entrenched interests, official narratives, and dominant institutions.

Historically, this ethos has been associated with the left, especially in traditions such as critical theory, “counter-cultural” art, and activist scholarship. Today, it increasingly captivates the right as well, where an “anti-elite” politics paints right-wing intellectuals as brave dissidents exposing where real power lies in society: in the “liberal establishment”, the “regime”, “the Cathedral”, or sinister networks of the “deep state”.

It’s easy to understand the ethos’s appeal. The world contains oppression, exploitation, and extreme power inequalities, many of which are unjust and harmful. Societies can’t rely on the powerful to check themselves. They will spread, fund, and amplify self-serving propaganda. So, intellectuals surely have a responsibility to push back against such falsehoods—to expose deception, unmask mystifying ideologies, and reveal what is really going on.

It’s easy to think of cases that fit this model of courageous intellectual activity: Mary Wollstonecraft challenging the subordination of women, Ida B. Wells documenting lynching in the American South, Solzhenitsyn exposing Soviet oppression, the reporters who broke Watergate or decades of Catholic Church abuse, the scientists who exposed tobacco industry propaganda, and so on.

Nevertheless, I will argue that in practice, the widespread embrace of this ethos among large segments of the Western intelligentsia is often harmful and counterproductive. It encourages intellectual laziness and self-deception in ways that undermine its stated aim.

To hold power accountable, societies need access to trusted truths about what is happening. At least in modern liberal-democratic societies, these truths are often highly complex and counterintuitive. Determining what they are is challenging. The ethos of speaking truth to power encourages intellectuals to think that such truths have been established before inquiry has even begun. It replaces a difficult epistemic task—finding out what is true, where power lies, and how truth and power interact in specific cases—with the simpler, more self-flattering goal of summoning intellectual courage.

Of course, intellectual courage is extremely important. But without first carefully establishing what’s true, it’s either pointless or harmful. So intellectuals’ primary responsibility is simply to seek and speak the truth, full stop. This job description is less heroic, but it’s more intellectually and socially useful, and it’s less vulnerable to self-deception precisely because it’s less heroic.

At least, that’s what I will argue here. Specifically, I will argue that an ethos of speaking truth to power has three problems: it prejudges what inquiry is supposed to discover; it licenses motivated reasoning under a heroic self-image; and it exempts intellectual elites from the suspicion they direct at others. For these reasons, it also threatens public trust in the institutions that democratic societies depend on to make sense of society and hold power accountable.

I will illustrate these arguments with several examples: the strange dogmatism of “critical” theory, the left’s failure to grapple seriously with AI, and the modern right’s collapse into conspiracism.

First, though, it will be helpful to start with a more general framework for understanding what power even is, and how it relates to the domain of ideas.

Subscribe now

The Causes and Complexities of Social Power

Historically, many intellectuals assumed that different kinds of power could be reduced to one fundamental kind. The canonical example is Marx’s view that economic power relations are upstream of all others.

As the sociologist Michael Mann argues in his series The Sources of Social Power, this reductionist perspective seriously misrepresents human societies. Power, Mann argues, has at least four distinct sources: political, economic, military, and ideological. Although they often reinforce one another, the connections are complex and “promiscuous”, and no power source can be reduced to the others.

Nevertheless, if we focus on the distribution and character of power in recorded human history prior to the emergence of modern liberal democracies, two broad patterns emerge.

First, most power relations were illegitimate and extractive. Although power took different forms and shifted among different elites, the elites and institutions they fought over were highly exploitative. Second, ideological power was typically harnessed to support and legitimise such extractive social orders. Hence Marx and Engels’ famous observation that “the ideas of the ruling class are in every epoch the ruling ideas.”

This “dominant ideology thesis” is an exaggeration. Even in the most extractive regimes, ideological power is typically at least partly independent of other sources of power and shows a surprising tendency to be unleashed and to disrupt existing social orders (Christianity, Islam, Protestantism, liberalism, fascism, communism, and so on). Moreover, the masses often see through elite ideologies, even if they pay lip service to them to avoid trouble.

However, as with most of Marx’s ideas, the thesis contains a grain of truth. Throughout most of recorded history, extractive elites conscripted and coerced ideological elites (priests, theologians, intellectuals, artists, and so on) to produce and distribute legitimising myths. These myths then dominated society’s formal communication channels.

In that kind of world, an ethos of speaking truth to power can serve as an understandable guide for brave, public-minded intellectuals. Because illegitimate power structures are relatively easy to identify and ruling ideologies are systematically false and self-serving, intellectual courage and accuracy often overlap. When a regime depends on enforced lies, refusing to go along with those lies is a contribution to the truth. When exploitation is naked, the main challenges public-minded intellectuals confront are practical and moral—being killed, imprisoned, or silenced—not epistemic.

Both today and throughout history, the courageous writers, reporters, intellectuals, and artists who have spoken truth to power in such regimes often deserve deep admiration and respect.

The problem is that the Western intelligentsia no longer lives in that world.

The Sources of Social Power in Open Societies

Today, Western intellectual elites live in liberal democracies characterised by universal rights, formal equality, the rule of law, pluralism, constitutional limits on state power, and significant political and economic freedoms. We also live in an era of unprecedented material prosperity, in large part a consequence of how economic freedom and competition are channelled into productivity and innovation through free markets, and political freedom and competition are channelled into social welfare, insurance, and public goods through democracy.

In these societies, social power takes on a very different form.

First, although nobody should think that power is always distributed in fair or functional ways in liberal societies, it’s even more misguided to treat all power asymmetries as illegitimate or harmful. Consumer choice and voting mean that political and economic power is more evenly distributed, and disparities in wealth and political influence often reflect not extraction but fair, socially beneficial processes. For example, wealth is often downstream not of theft or rent-seeking but of free markets that reward innovation and efficiency, benefiting everyone. Similarly, political power is typically allocated not by brute force but through complex processes of democratic participation or the broadly meritocratic selection of civil servants.

Again, the point is not that there is no corruption, exploitation, rent-seeking, or domination in modern liberal societies. There is a lot. The point is simply that allegations of such things must be carefully adjudicated on a case-by-case basis. They cannot simply be assumed in advance of inquiry.

Second, ideological power operates very differently in liberal-democratic societies. As figures from Karl Popper to Jonathan Rauch have argued, open societies are distinguished as much by their free, competitive, and pluralistic epistemic characteristics as by their political or economic systems.

One aspect of this concerns the emergence of imperfect but real epistemic freedom for the first time in human history, including freedom of belief, free speech, a free press, and academic freedom. Open societies lack extensive top-down regulation of the information environment, enabling a vibrant, pluralistic public sphere and marketplace of ideas. Another, related dimension concerns the deliberate creation and funding of epistemic institutions, including news organisations, universities, fact-finding agencies, and more. Whether these institutions receive state or private funding, they are typically defined by their independence from other power structures in society and derive public legitimacy and support precisely from such neutrality.

Again, the point is not that in liberal-democratic societies ideological power floats completely free of other power sources. As I have argued elsewhere, the liberal “marketplace of ideas” often functions as a de facto marketplace of rationalisations, and political and economic elites often have greater “purchasing power” in such markets than ordinary people.

Nevertheless, a simple picture in which ideological elites are mere hand-puppets of other elites is a gross distortion. In liberal societies, ideological elites wield substantial independent power and are often far more influenced by their own norms, fashions, and status games than by top-down political authority or economic influences.

In fact, in a clear trend that has accelerated since at least the 1960s and the birth of a powerful “counter-culture” prestige economy, much of the intelligentsia and art world defines itself in explicit opposition to established political and economic power centres. Hence the widespread appeal and embrace of an ethos of speaking truth to power. And today, of course, the media environment in liberal societies is characterised by strong demand on both the left and right for populist denunciations and condemnations of establishment institutions. Critiquing the powerful can function as a lucrative source of cultural esteem and financial rewards.

Three Problems with “Speaking Truth to Power”

Under these conditions, a governing ethos of speaking truth to power no longer makes practical sense for intellectuals.

1. The Replacement Problem

First, if the ethos is to function as a guide to action, the intellectual must already know three things: what the truth is, where power lies, and how truth and power conflict in specific cases. These are precisely the things that intellectual work is supposed to establish, however. So, in practice, the ethos typically puts the cart before the horse, replacing challenging epistemic labour—figuring out what is actually happening in societies characterised by profound ambiguity, uncertainty, and complexity—with a much simpler moral posture.

Consider the following questions. Are experts defending genuine knowledge or protecting their institutional status? Is growing populist distrust of elites a rational response to establishment failure or a symptom of pervasive misinformation? Does social media democratise public debate, degrade it, or merely reveal pre-existing conflicts? What is the empirical track record of “neoliberal” policies? Did globalisation fuel anti-immigrant sentiment across Western countries? Is AI just another “normal” technology or is it on track to trigger an “intelligence explosion” that will upend our economy, politics, and culture?

Answering these and countless other questions requires patient, careful inquiry and data collection, rigorous social-scientific methods, intellectual humility, and deliberate attempts to seek out disconfirming evidence and dissenting viewpoints. A simple heuristic that power and truth systematically collide doesn’t get you anywhere. Powerful people often speak the truth; the powerless often speak nonsense; and very often nobody—neither the powerful nor the powerless—has a clue what is going on, which is why we need rigorous, trustworthy, truth-seeking epistemic institutions in the first place.

The Left’s Embrace of Sophisticated Conspiracy Theorising

In many ways, the clearest illustration of the replacement problem lies in “critical theory” and much of academic “critical studies” in modern universities. Although this work comes in many different forms, its unifying commitment is to some version of Marx’s dominant ideology thesis: that the responsibility of intellectuals is to unmask and oppose ruling (“official”, “hegemonic”) ideologies (“discourses”, “regimes of truth”, etc.), thereby emancipating the downtrodden and oppressed from the illusions that sustain their subordination.

In other words, it is to speak truth to power.

Although there is lots of interesting and insightful work in the broad tradition of critical theory, Joseph Heath is right when he observes that the most striking thing about critical studies today is how thoroughly uncritical—how dogmatic—it tends to be. Much of its intellectual energy and activity proceeds on the basis that the fundamental truths about society have already been established. It is simply taken for granted that debunking, suspicion, and unmasking of designated villains (capitalism, racial capitalism, neoliberalism, neocolonialism, and so on) is valuable. “Research” proceeds primarily by applying and extending this worldview, typically through the hermeneutic parsing of sacred texts (Marx, Adorno, Gramsci, Foucault, Fanon, Butler, and so on), rather than through rigorous social-scientific analysis that leaves open the question of whether the worldview is actually accurate or applicable in specific cases.

In fairness, the intellectual problems with this approach have gradually dawned on some within this tradition, especially once it became clear that the political right could simply appropriate the same toolkit to demonise the left intelligentsia and discredit the causes it champions. (More on this below.) Unsurprisingly, for example, the fashionable idea that all knowledge is socially constructed and entangled with power lost some of its appeal once climate activism became a defining left-wing cause. The analysis is not exactly helpful for encouraging people to “Trust the Science”.

As Bruno Latour belatedly acknowledged, a totalising hermeneutics of suspicion directed towards “power” and “dominant institutions” is no better than high-IQ conspiracy theorising: it simplifies complex realities, flattens important distinctions, and protects itself against refutation by dismissing all criticisms as predictable responses from those in power.

Similar lessons generalise to activist scholarship more broadly. When scholarship is guided by activism—in universities today, that mostly means “social justice” activism, but the point generalises—scholars assume that they have a clear and accurate sense of what justice requires. Their job is merely to “advocate” for justice through their research. Although this work can be valuable, it too often encourages a kind of cosplay of intellectual activity, in which the goal is not to figure out what’s true but to recruit any intellectual ammunition (evidence, facts, arguments, testimony, interpretations) available to support and extend a predetermined political vision. The question of whether the vision is actually correct or applicable in a given case is settled before inquiry has even begun.

Of course, these problems might not matter if they were confined to esoteric corners of humanities departments. But the core ethos driving these problems now shapes large bodies of work produced among the left-wing intelligentsia and pundit class.

Subscribe now

The “AI Con” Con

Consider artificial intelligence. In the space of a few years, AI systems have gone from barely producing coherent text to achieving expert-level performance across a vast range of benchmarks, attaining superhuman coding abilities in many domains, and increasingly behaving like general-purpose, tool-using, agentic assistants. Across countless distinct evaluations and metrics, their capabilities have improved at a staggering exponential rate. For these reasons and more, AI progress is now backed by one of the largest capital expenditures in human history, and Anthropic, a frontier AI company, may be the fastest-growing company in the history of capitalism.

These developments throw up many difficult questions. What should we make of modern AI’s capabilities? As these systems improve across many aspects of AI research and development, are we on the cusp of a process of “recursive self-improvement” in which AI systems automate the task of producing new AI systems, fuelling explosive, rapid technological growth? How will we control this technology as it becomes more powerful? How will it impact and disrupt the labour market, our political institutions, culture, and humanity’s sense of meaning and purpose?

These are challenging questions. There are profound expert disagreements and ambiguous, contested evidence across many fields. And yet, in the face of these challenges, large segments of the left-wing intelligentsia and commentariat have coalesced around a simple viewpoint: that AI is basically a nothingburger, a “con”; that state-of-the-art large language models are little better than “fancy autocomplete” or “stochastic parrots”; and that anyone who claims otherwise, including experts who have long worried that AI is extremely dangerous and will pose existential risks, is simply part of an elaborate propaganda machine designed to over-hype a product being sold by deceptive AI companies.

This attitude is certainly not universal among the left, especially as its collision with reality is becoming increasingly clear. There are thoughtful left-wing analyses and growing attention to the topic among influential leftist politicians, including Bernie Sanders. Moreover, there is reasonable disagreement about how powerful AI systems actually are and are likely to become, how informative benchmark-based exponential trends are, and whether AI should be thought of as a sui generis, abnormal technology.

Still, it’s difficult to overstate the scale of the intellectual failure that has occurred among many left-wing intellectuals here—and, by extension, the many departments in universities where they dominate—which has gone far beyond responsible scepticism into the domain of outright denial. This failure is especially galling because a thoughtful, reality-oriented, left-wing intellectual culture would have much to contribute to many of the issues raised by highly capable and rapidly advancing AI, including oligarchy, concentration of power, economic inequality, and labour automation.

As Will MacAskill observes, it is striking that in the midst of a sustained attempt by some of the most powerful capitalists in human history to build a technology openly designed to replace human workers—something that Marx himself literally predicted would happen—the Bluesky intelligentsia has pre-decided it is all hype and propaganda.

What happened? Obviously there is no single answer to this question, but one part of the answer is how much of the modern left’s self-identity of “speaking truth to power” has replaced intellectual inquiry with the application of a predetermined template. In this case, the application of the template runs: Powerful “tech bros” say AI is powerful; The claims of capitalists can be dismissed as self-serving propaganda; So, AI is not really powerful.

There’s no need to check whether this template is actually applicable in this specific case, or to every detail of the case. The core issues have been decided in advance. Indeed, many on the left take pride in refusing to even use AI systems like ChatGPT, even when their literal job is to write about or research them, and even when they make confident claims about what these systems cannot do. The role of intellectual inquiry is to take the predetermined template and find ways of formulating, extending, and rationalising it.

The problem here is not just the bad epistemics. The deeper problem is that the bad epistemics are self-defeating. If your goal is to speak truth to power, it’s important to first carefully determine what is actually true. If you don’t do that, you undermine your goal by misrepresenting the challenges in ways that discredit critics among anyone with actual expertise on the topic. You make power less accountable, not more so.

2. The Motivated Reasoning Problem

There is a second, related problem that arises when intellectual elites embrace an ethos of speaking truth to power: it fuels corrosive forms of motivated reasoning. It doesn’t just let a pre-existing worldview shape inquiry; it tends to systematically distort that inquiry, giving favoured conclusions a much easier pass than unfavoured ones.

Of course, motivated reasoning is a universal tendency. But whereas people should generally be embarrassed by it, an identity of speaking truth to power tends to reframe it as a virtue.

First, it casts intellectuals as brave truth tellers, opposing illegitimate power on behalf of the powerless. Once one identifies with this self-image, isolated demands for rigour no longer seem like a failure of rigour. They look like bravery. Courageous, truth-seeking intellectuals doubt the powerful; only cowards and apologists for power doubt the powerless. Second, the ethos creates a status game where prestige flows to those who unmask and oppose power, and anyone who complicates or challenges those critiques is viewed with suspicion or outright contempt.

In intellectual environments that operate according to this ethos, the result is an informational ecosystem in which claims that demonise villainous elites and institutions are credulously accepted and amplified, whereas any claims that cast them in a positive light are scrutinised, ignored, or dismissed. Once again, this might not be such a problem in societies in which all power is nakedly illegitimate and extractive. But in contexts involving much greater complexity and opacity, it is intellectually poisonous.

This dynamic is on steroids in the AI debate. In Karen Hao’s highly influential, award-winning book Empire of AI, for example, possibilities that many of the world’s leading AI experts take extremely seriously—that we might soon build general-purpose, autonomous, super-intelligent AI—are simply dismissed as “a fantastical, all-purpose excuse for OpenAI to continue pushing for ever more wealth and power.” At the same time, she reports wildly inaccurate claims about AI water use. (Hao has since corrected these mistakes, which were identified by Andy Masley.)

This is a general pattern in much left-wing commentary on this topic. Claims that might suggest frontier AI systems are actually highly capable, useful to many people and businesses, or carry societal benefits (e.g., in science, healthcare, and education) are dismissed as industry hype, even when they come from independent experts. Even claims that such systems have extremely dangerous capabilities are dismissed as further hype because they imply the systems in question are genuinely capable in the first place.

This extreme scepticism and cynicism then suspiciously vanish when confronted with any claims that cast modern AI in a negative light, often no matter how misinformed or dubious they are. (See, for example, prominent but highly misleading claims about AI and the environment, AI and racial discrimination, AI and misinformation, or, the strange, continued insistence that AI systems can’t do things that they can demonstrably do.)

The motivated reasoning problem is that these highly asymmetric standards aren’t recognised as an intellectual failure. They are experienced as heroic activism.

The Right’s Embrace of Unsophisticated Critical Theorising

These problems are not unique to the left. In the last couple of decades, large parts of the right have adopted the identity of anti-establishment politics, embracing much of the aesthetics, rhetoric, and cognitive style of left-wing critical theory and counter-culture. Once progressives came to dominate establishment institutions, the right became the anti-institutional faction that views itself as speaking truth to power. As Joe Rogan once put it, “The rebels are Republicans now. You want to be punk rock? You want to like buck the system? You’re conservative now.” (In support of this assessment, The Sex Pistols’ John Lydon is now a MAGA Trump supporter.)

This anti-establishment, counter-cultural energy has fuelled the demand for a new right-wing intelligentsia—I use the term loosely—capable of articulating and rationalising the core worldview. So, gone are the days of serious intellectuals like Friedrich Hayek, Milton Friedman, Thomas Sowell, or Roger Scruton. For much of the modern right, the intellectual energy now resides with writers, pundits, and social media posters such as Curtis Yarvin, Christopher Rufo, Peter Thiel, Tucker Carlson, and Elon Musk.

Despite superficial differences in how these figures view the world, the core worldview is similar: that real power in the modern world lies not with ordinary political, economic, or military elites but with the coercive ideological power exercised by established institutions such as legacy media organisations, universities, government agencies, and so on. This is “the Cathedral”, the “regime”, the “deep state”, etc., which enforces a stifling and destructive progressive orthodoxy.

There is, of course, some irony here. When the ideal of a counterculture and critical theory began to gain popularity in Western universities in the 1960s, the “system”—the enemy—was capitalism, capitalist elites, and social conservatives. Now, pretty much the same worldview (and sometimes even the same intellectual gurus) has been systematically inverted by a powerful new political movement for whom—as J.D. Vance has put it in a title of a 2021 speech—“the universities are the enemy”.

Just as with the traditional left’s critique of capitalism, there are some important grains of truth in the anti-establishment, “dissident” right’s critique today. It’s simply true that progressive groupthink is too common in many establishment institutions, and that the cultural values and priorities of the professional-managerial class that staffs these institutions play a powerful role in modern Western societies. In Mann’s vocabulary, cultural progressives wield extraordinary ideological power in the modern West, and this power is worthy of scrutiny and critique.

Nevertheless, the right intelligentsia’s new self-image as speaking truth to power exhibits even worse intellectual pathologies than those that have long characterised left-wing critical theory. In place of careful, evidence-based investigations into universities or the administrative state, these institutions are reduced to simplistic, stick-figure caricatures aimed at homogenising and demonising them in extreme ways. Any evidence that supports such caricatures, no matter how cherry-picked or misinformed, is amplified, and any evidence that complicates or challenges them is either dismissed or ignored.

Leftist Indoctrination Camps

Consider universities, for example. Universities in the US and other Western countries are highly complex institutions. They often house thousands of highly trained researchers working on many distinct scientific and intellectual projects and oversee a diverse curriculum across subjects such as quantum mechanics, macroeconomics, cancer research, and personality psychology. Unsurprisingly, given the sheer amount of intellectual ingenuity and research there, they have also been a major force for scientific and technological progress in the modern world.

As Steven Pinker observes of Harvard, although there are undeniable issues with progressive groupthink and free speech in the university, most faculty don’t identify as “very liberal” or on the “radical left”, and there is considerable intellectual diversity and disagreement in terms of research, the personal views of researchers, and what gets taught. Only a small minority of courses with relatively small enrolments are explicitly “woke”, whereas the most popular undergraduate courses tend to be in fields like economics and computer science.

In other words, despite the institution’s obvious imperfections, it holds immense value, and it is a highly intricate system that defies simplistic generalisations. And yet, if you read the “dissident” right-wing intelligentsia, facile slogan-based demonisation is pretty much all you get, with status flowing to those who denounce Harvard and other institutions in the most hyperbolic ways possible—as a “national disgrace”, a “woke” or “Maoist indoctrination camp,” and so on—all dressed up as “based” denunciations among those brave enough to be “red-pilled”.

“Gay Race Communism”

Consider Curtis Yarvin, for example, a tech entrepreneur-turned-intellectual who coined the terms “red pill” and “the Cathedral” as they are used today, and whom J.D. Vance has cited as an intellectual influence.

In recent reflections on why he supports the Trump administration and its assault on many establishment institutions, Yarvin declares that the governing “regime” has “invented a pandemic and killed 20 million people, for absolutely no sane reason at all”, and “has spent the last century managing public opinion with every carrot and stick it can find, up to and including asking professors to compose their own inventive and detailed loyalty oaths to gay race communism.”

As with the rest of Yarvin’s intellectual outputs, it’s challenging to overstate how low-quality these arguments are. Not only is there no monolithic governing “regime”, but the existing evidence for the lab leak hypothesis is highly mixed; almost all the actual evidence we have comes from the very scientists, intelligence agencies (the “deep state”), and legacy media reporters treated as part of this “regime”; it was Donald Trump’s first administration that ended an Obama-era pause on funding high-risk gain-of-function research in 2017; whereas establishment scientists and academics wrote critiques of such research, MAGA supporters and “dissident” right intellectuals didn’t; it was scientists who invented vaccines that saved millions of lives, along with countless other medical and technological advances over the past century; and the framing of DEI statements (in my view, a genuinely objectionable practice) as “loyalty oaths to gay race communism” is a preposterous exaggeration.

In a functional intellectual culture, this sort of analysis would be a source of extreme embarrassment and shame. But for Yarvin and other pundits and writers operating within the “contrarian” right-wing status economy, they are experienced as bravery—as part of a red-pilled escape from being a “libtard and a coward”.

This is what happens when the task of intellectual inquiry is replaced by a governing ethos of speaking truth to power. It allows charlatans and know-nothings to reframe intellectual malpractice as a virtue.

Subscribe now

3. The Self-Exemption Problem

The example of the modern “dissident” right also highlights a final, obvious problem with this ethos: many of the movement’s central figures are millionaires or billionaires who wield immense power and considerable influence over the governing administration of the world’s most powerful country.

On the one hand, this illustrates why this intellectual movement’s dysfunctional epistemics are so costly. For example, the modern right’s lazy, conspiracist worldview has had countless terrible real-world consequences, not least in the millions of lives projected by some experts to be lost or put at risk by Elon Musk’s inept DOGE cuts on USAID.

However, it also illustrates a deeper flaw with how the ethos of speaking truth to power operates in practice. By its very nature, this self-image exempts intellectual elites from the suspicion they direct at others. It implicitly treats the intellectual as either powerless or a defender of the powerless.

Although the absurdity of this is especially salient in the case of the modern right-wing intelligentsia and pundit class, it also applies to large swathes of the left-wing Western intelligentsia, whose disproportionate influence in universities, legacy media outlets, and the art world grants them considerable ideological power.

As Mann’s analysis of social power reveals, such ideological power—the power to shape ideas, cultural fashions, and norms—is real. This is why modern culture wars—conflicts over norms, ideas, and symbols—are so heated, and why those who already wield considerable economic power (e.g., Musk, Peter Thiel, Marc Andreessen, and so on) are so obsessed with converting it into ideological power, often through incessant social media posting. If ideological power were merely derivative of economic power, these efforts wouldn’t be necessary.

The existence of ideological power raises awkward questions for those who wield it under the banner of speaking truth to power. If the ethos’s animating insight is that the powerful often embrace self-serving, self-aggrandising narratives, shouldn’t we also turn such suspicion towards intellectual elites themselves? Mightn’t the ethos itself function as a kind of legitimising myth, a way of dressing up activity often rooted in grubby motives—self-aggrandisement, status competition, demonising rivals, and so on—in suspiciously noble clothing?

Once again, this worry is not pressing for true dissidents fighting clear cases of oppression and exploitation. Those who risk life, limb, or reputation to speak truth to power in such cases do so from a position of subordination and personal risk. This is what makes the behaviour so admirable.

For much of the Western intelligentsia today, however, this is simply not the situation. For Ivy League professors, New York Times journalists, the podcast class, or Hollywood actors, “speaking truth to power” is often met not with crushing opposition from a brutal regime but with applause, approval, and accolades.

If we should be highly suspicious of power and its self-serving propaganda, why do intellectuals expend so little effort turning this suspicion inwards on ourselves? (There are some excellent exceptions here.) If power corrupts cognition—a genuine insight—then so, presumably, does ideological power, along with the status games through which it is allocated in universities, journalism, media, and the arts.

Of course, one might respond that this kind of view is too cynical, and that the mere fact that intellectual elites might have impure motives doesn’t show that their claims are wrong. And I agree. We shouldn’t pre-judge that the powerful’s claims are self-serving propaganda before inquiry has even begun—before we’ve undertaken the hard work of figuring out what is actually true.

In other words, an ethos of “speaking truth to power” is bad epistemology.

In this episode, Henry and I discuss these and many other questions with David Kipping, Associate Professor of Astronomy at Columbia University, where he leads the Cool Worlds Lab. David’s research spans exoplanets, exomoons, Bayesian inference, technosignatures, and the search for life and intelligence beyond Earth. He is also one of the best science communicators working today through the Cool Worlds YouTube channel and podcast.

Subscribe now

Among other topics, we discussed:

• David’s Red Sky Paradox: if most stars are red dwarfs, and red dwarfs live for vastly longer than stars like the Sun, why do we find ourselves orbiting a yellow star?

• Whether anthropic reasoning — reasoning from the fact of our own existence — is a profound scientific tool, a philosophical minefield, or both.

• The reference class problem: when we reason about “observers like us”, who or what exactly counts as being like us?

• The Doomsday Argument, and why some apparently bizarre forms of probabilistic reasoning can nevertheless be powerful.

• The Fermi Paradox: if the universe is so large, and if life or intelligence is not fantastically rare, why don’t we see clear evidence of extraterrestrial civilisations?

• Whether advanced civilisations would spread through the galaxy using self-replicating probes — and why the absence of such probes might be one of the strongest constraints on extraterrestrial intelligence.

• How recent developments in artificial intelligence affect the Fermi Paradox. If humanity is close to building systems that can massively accelerate science and engineering, shouldn’t someone else have got there first?

• Whether artificial intelligence makes the simulation argument more plausible.

• David’s experience using artificial intelligence in scientific research, and why a meeting at the Institute for Advanced Study changed how he thinks about the role of these tools in science.

• Why David thinks artificial intelligence already has something close to “coding supremacy”, but is still far from being able to do science autonomously.

• The risks of AI-generated scientific slop: papers, peer review, and training data polluted by low-quality machine outputs.

• Whether artificial intelligence will make science more productive, or instead strip it of some of its deepest human value.

• Why the future of science communication may depend on better collaboration between academic institutions and independent creators.

Links and further reading

1. Cool Worlds Lab — David’s research group at Columbia University, focused on extrasolar planetary systems, exomoons, habitability, technosignatures, and related questions.

2. Cool Worlds on YouTube — David’s excellent science communication channel, covering astronomy, exoplanets, alien life, the Fermi Paradox, cosmology, and much else.

3. Cool Worlds Podcast — David’s podcast, featuring conversations on astronomy, technology, science, engineering, and related topics.

4. Cool Worlds Podcast: “We Need To Talk About Artificial Intelligence” — the solo episode in which David reflects on artificial intelligence and science after a meeting at the Institute for Advanced Study.

5. David Kipping’s Columbia profile — short institutional profile with background on his research.

Transcript

• Please note that this transcript has been lightly AI-edited and may contain minor mistakes.

Henry Shevlin: Welcome back. Our guest today is David Kipping, Associate Professor of Astronomy at Columbia University, where he leads the Cool Worlds Lab. His research spans exoplanets, exomoons, and the search for extraterrestrial life and intelligence, and he brings a Bayesian rigor to questions that could easily drift into speculation. He’s also one of the best science communicators working today with over a million subscribers on his Cool Worlds YouTube channel, where I should confess, I’ve spent an embarrassing number of hours watching when I probably should have been doing philosophy of AI.

David, like many of the best people, is a Cambridge alumnus, although unlike us, he actually studied something useful, namely natural sciences, before going on to do his PhD at UCL and postdoc at Harvard on the Sagan Fellowship. His work also has a really fantastic philosophical dimension, particularly around anthropic reasoning and observation selection effects, which makes him a perfect guest for two cognitive scientists who are finally getting to talk to an actual scientist. So David, welcome to Conspicuous Cognition.

David Kipping: Thank you for that very generous introduction.

Henry Shevlin: This is a bit of a fanboy moment for me, for real though. I really have spent like hundreds of hours at this point on Cool Worlds. But I’m going to get past it. I’m going to be a serious host.

David Kipping: It’s always weird when people say that to us, because I just imagine no one watches them. If it gets in my head that people are watching them, I’ll get tightened and anxious about what I’m saying. I just imagine I’m talking to a brick wall or something, and that’s much easier.

Henry Shevlin: Honestly, half the Warhammer figures in this room were painted while I was listening to Cool Worlds. I’ll leave it at that. Maybe a good place to start would be discussing anthropic reasoning, since that’s a real natural intersection at the boundary of astronomy and philosophy. Could you just give us a brief view of how you see anthropic reasoning, and maybe tell us a little bit about the Red Sky Paradox, which is one of your distinctive contributions to this area?

Anthropic Reasoning and the Red Sky Paradox

David Kipping: Yeah, I think one of the most interesting data points when it comes to asking questions about the search for life in the universe and our own place in the universe is our own existence — just the fact that we’re here. Anthropic reasoning has in many ways really been born out of cosmology. Cosmology had a rich history of using this. I think one of the first successful examples was by Steve Weinberg, a cosmologist who’s really a giant in the field. I think he’s now passed away, but he showed that you could predict not only the existence of the cosmological constant, but also its value to within a factor of a few, just based off of anthropic reasoning.

The argument was something like: the cosmological constant causes the universe to expand. It’s what causes the accelerating expansion of the universe. And so if you make that number too large, then structure would not form in the universe. You couldn’t form galaxies because everything would just fly apart too quickly. And if you make that number too small, or even negative, then you’d cause everything to recombine too quickly. So there has to be some Goldilocks value in order to explain our own existence. And so he predicted that.

At the time, the cosmological constant was kind of even a controversial idea — that it should exist because, obviously, Einstein’s general relativity, there’s that whole history of it being like his greatest blunder, of whether that should really be in there or not. People were kind of thinking that could be a static universe, and he predicted it successfully. So that was a really powerful use of it. And then Brandon Carter was the one who really kind of championed it and used it in all sorts of contexts.

In recent years, I’ve been thinking about it in an astrobiological context — how can we use it to ask questions about life in the universe especially, and our place in it?

For the Red Sky Paradox in particular: one interesting curiosity that seems to violate the norms of probability. The norms of probability would be to say that if there’s a Gaussian, a bell curve of possibilities, you should expect really to be near the center of that bell curve. It would be kind of weird if you lived many, many sigmas, many, many standard deviations off to the outside, either negative or positive direction. You’d expect to be somewhere in the middle. We sometimes call it the mediocrity principle, or something like this.

If you look at stars in the universe, most stars in the universe are red dwarfs. About 80%, 82% of stars are red dwarfs, which are stars less than half the mass of our own sun. So they’re very, very numerous. They’re called red dwarfs, of course, because they’re so low mass — they don’t have the internal pressure, the gravity, to fuse as much energy as the sun does. And thus they have less luminosity, and so their temperature is cooler. That’s why they look red.

Not only do these stars have this 80%-plus frequency — Sun-like stars are something like 6%, I think, frequency, an enormous ratio, just straight off the bat, about 30 to one or something — but on top of that, they live really long. These stars live for trillions of years potentially, especially the lowest mass ones. And so if you flash forward into the future, tens of billions of years, hundreds of billions of years, there wouldn’t be any Sun-like stars left, really. There’d be very, very few of them. And the only stars that would be glowing would be these red dwarfs.

So if you ask yourself — and this is sort of called the strong self-sampling assumption by Nick Bostrom, where you allow yourself to be born at a random moment in time — if you were born at a random moment in the history of the universe, then the advantage of the longevity of these red dwarfs really manifests. It ends up being more than a thousand to one odds that, if you’re a random soul, a random observer born around either a red dwarf or a yellow dwarf, you’re much more likely by over a thousand to one — I think 1600 to one — to be born around a red dwarf.

So I call that the Red Sky Paradox because it’s just odd. If all things being equal — and that’s kind of the base assumption there, that red dwarfs are just as good for life as Sun-like stars — you might question that assumption. That’s always the point of a paradox: a paradox shows a logical contradiction that then you can revisit the assumptions under which that paradox was derived and say, one of those assumptions must be wrong.

So for the Fermi paradox, you might say, if life is everywhere, how can we not see anyone? Therefore, the assumption to revisit is that life is everywhere. And here, with the Red Sky Paradox, we might challenge the assumption that red dwarf stars are even capable of sustaining — and really specifically — complex life like us, observers. Maybe they have simple life, but something prohibits them from evolving all the way through to something that can do statistics, do astronomy, do geology — like learn about its planet and kind of essentially write the paper that I wrote about the Red Sky Paradox. That’s kind of the cogito ergo sum criterion I’m using as my conditional in this reasoning.

I have been making that suggestion to colleagues because the James Webb Space Telescope right now is heavily invested on red dwarfs. There’s a good reason for that. It’s kind of all they can do. Unfortunately, it just doesn’t have the capability, the technology, really to do anything with Sun-like stars. But red dwarfs, it’s game on. And I’m just saying, look, there might be reasons why it won’t turn up anything.

Henry Shevlin: And is the specific suggestion, I think I’ve heard, that basically in the early years of the early formation of red dwarf stars, they might be especially turbulent in a way that sort of scorches any planets in their vicinity and strips away their atmospheres? Is this one of the empirical predictions that we can make on the basis of the Red Sky Paradox?

David Kipping: I would say it’s more a consistency than a prediction. I try to be very careful. I love very broad agnostic reasoning as much as possible. In this case with the Red Sky Paradox, I don’t have to invoke any mechanism specifically. There is probably a mechanism, surely there is a mechanism — unless we are really a one in 1600 outlier. That’s possible as well, and I concede that it is possible, that we are just a very unusual example.

But if that’s not true — for a typical example — then there is some mechanism which bars the evolution of observers like ourselves. And in the paper, I point out there are numerous mechanisms people have suggested, including the fact that these stars have large coronal mass ejections coming off them, which can strip planets of their atmospheres. They have a prolonged, what we’d call adolescence for a star. Our Sun went from being born to being a main sequence star in the space of about a hundred million years, even less than that, tens of millions of years. Whereas red dwarfs take a billion years sometimes to settle down. And during that adolescence phase, they’re violent, and they can actually remove all the water off their neighboring planets.

We think it’s that, that’s when water gets delivered. Our water was probably delivered by comets during the late heavy bombardment and the other bombardments that were occurring before that. And so if during that time you’re delivering water through comets, the comets get depleted, but the star is so active it’s stripping all the water off them, then you’re kind of net zero — like you don’t end up with any water at end of the day. And then, when all said and done, you’ve just got dry planets around a normal star, but it’s too late. There’s no more water left to deliver to the planet anymore. So that’s been suggested as well.

Then there’s also the questions about photosynthesis. Is photosynthesis possible if the star is much redder than our own star? Because obviously plants on Earth use blue light as well as red light. If you take away all the blue light, how will they do? We don’t know. It’s kind of unclear. We don’t really have too many examples of life on Earth which thrives under those conditions. And then there’s tidal locking — these planets have probably one side of the planet facing the star.

So there’s many sensible concerns. But what I’m trying to do is avoid saying it’s this, it must be this one. Because that’s really for the astrophysicists studying the geology of those objects to figure out. I’m just saying there probably is something, and go after it.

Dan Williams: I’m not sure entirely how to frame this question, David, but someone might respond that there’s just something a little bit weird or surprising that you could draw seemingly substantive inferences from such a slim evidential basis. The starting observation here is just we exist where we do. And then there’s this interesting probabilistic reasoning. And then that’s leading you potentially to draw inferences about where life might potentially evolve in the universe. I suppose this is just an objection from the perspective of, isn’t there something a little bit weird about this entire style of reasoning?

David Kipping: It’s definitely weird. Yeah, it’s very weird. I always think Nick Bostrom is really like the father of all this kind of thinking in the modern era. And he often concedes that point, that it is very strange. We don’t really have a complete theory of anthropic reasoning. It’s sort of a work in progress, to some extent. In the same way, we don’t really understand how AI works. We don’t understand really the full nature of the universe. They are works in progress.

And yet it also seems logically, you can pose these logical questions that seem irrefutable or compelling. So like I mentioned the Weinberg example: it is really hard to imagine how you could possibly have the cosmological constant be a thousand times what it is, because Weinberg’s right — you just wouldn’t have galaxies. So how could you possibly have us in that situation?

The fine-tuning argument for the multiverse is the other popular use of it in modern science, I would say. They often point out, why is it that the gravitational constant and the fine-structure constant and the speed of light, all these things are just the way they are? There’s a simple anthropic reason for it. You don’t have to accept it, but you can certainly make this argument that if they were anything else, you wouldn’t be here to talk about it. So you can’t really change the mass of the electron by a factor of ten and get away with it. There’s going to be repercussions to chemistry. If you made the speed of light ten times slower than it really is, then relativistic effects happen in sort of everyday cases, especially for chemistry — that impinges the ability of electron shells to be stable. So you start to really ruin the CNO cycle inside stars and stuff. You start to ruin a lot of interesting nuclear physics and chemistry. So you can see, I think that’s the most common case.

There’s also a fun case — I think this is true, but it’s kind of a bit of an urban legend — that during World War II, there’s this thing called the German tank problem, if you’ve heard of that. The Allies would apparently — maybe you know better than I do whether it’s true or not — would see the numbers imprinted on German tanks. It would say one-five-five or something. So they would look at that number and say, okay, so they must have a border of like 300 tanks. Because if that’s a typical number, they’ve probably not got a million tanks, because otherwise it’d be very unusual that we had the 155th tank out of a million that were being produced. And they probably don’t have 155 tanks, because then we’d just be very lucky that we’d caught the very last tank that was manufactured. It’s probably of order three to 400. And so they used that to set the manufacturing constraints for the factories back in the UK — like, this is how many tanks you need to produce, because we think that’s how many the Germans have.

So yeah, there’s examples of this reasoning being used quite a bit. I think the one way it really troubles people is the doomsday argument. I think that’s kind of like the one that everyone gets — no, something doesn’t feel right about that when you apply it to that case.

Dan Williams: Could you walk through what the doomsday argument is, David?

The Doomsday Argument

David Kipping: Yeah, sure. So it’s been invented like three or four times, I think, by different people at this point. It essentially says that if we are a medium example of ranked humans that ever live — so you go from, I mean, this is where it gets a little bit, I always think, a bit ill-defined — like, you have somehow a first human who lived, I don’t know, a million years ago or something, and then you go all the way up to today, and maybe you count up that it’s of order of sort of 100, 200 billion humans who’ve ever lived throughout human history.

So if you’re somewhere in the middle, then you’d expect there to be about another 200 billion humans to go before we call it a day. And of course, the birth rate is much higher — there’s much more people than there is today, more importantly. So the number of absolute people that are being born is much higher than it ever has been in history. And so that means there’s probably only like five or six generations left, or something, before you run out of these people. And so that’s kind of disturbing because it implies that there’s only like a hundred or a few hundred years to go before doomsday will happen.

So a lot of people think that’s really weird. How could you possibly take your rank position and make inferences about the extinction of humanity? When it’s framed like that, I think it feels really flimsy. But on the other hand, if you frame it slightly differently — you look at like the Foundation series or Star Wars or something like that, where they have these galactic-spanning empires — and you think how many individuals must be living in those societies. They’re all humans, right? They’re humans just living all over these planets in the Foundation series. You’d have trillions and trillions and trillions of trillions of people. And the chance, if you were born as a random soul at a random time, that you would be on the progenitor planet, pre-empire phase, would be vanishingly small. So you might therefore make the argument that that doesn’t look like a likely future for us. It doesn’t seem likely that humanity will ever become a galactic or universal-spanning species, because how does that possibly make sense with us being so early in the story?

But there’s lots of ways to criticize it. One is that maybe humans change. Maybe the experience of a human in a thousand years from now is some kind of cyborg, or genetically modified version of us, or just natural evolution that — their experience is not the same as us. And so we can’t say that they’re a representative example. That’s kind of the key part of this assumption. You can draw a random member, but maybe the membership itself evolves in some subtle way.

And certainly that goes backwards in time, like, when does Homo erectus suddenly become human and suddenly not? It feels very artificial to draw a line. Do you include all animals that have ever lived by that metric? How does this work? So I think that’s where, when you start ranking people, it gets really flimsy. But I think this is more a criticism of the ranking aspect of the anthropic argument, and the anthropic reasoning itself. I think it’s more to do with the ranking — that it’s probably an ill-defined problem to try and rank and discretize people like this, because of the changes that happen to humans.

Henry Shevlin: So this is one thing that I get hopelessly confused about when I think about anthropic reasoning, which is sort of the reference class problem. How do you decide how to specify your sample? Because in the case of the Red Sky Paradox, you might say, well, I step outside and I see a yellow star, right? So of course it’s impossible that I could ever have been born around a red star. So you could condition the reference class on the type of observers living under yellow star atmospheres. Why doesn’t that diffuse the problem?

David Kipping: Well then you’re kind of like double conditioning. You’re almost like saying, what’s the probability of having water on your planet given that you have water on your planet? Well, it’s one. I mean, obviously it’s one, because it’s a double, it’s self-conditional, it’s a circular statement. Obviously you can certainly make such a statement, but it doesn’t teach you anything. So you can say, what’s the probability of having a yellow Sun given you have a yellow Sun? But it doesn’t move the needle in any way.

So you do have to make a stretch. And so that stretch here would be: what’s the probability of an observer seeing a yellow star under the assumption that observers are equally likely to be born around any type of star, or any main sequence star, to be a bit more specific? So that’s the tacit assumption. And it’s reasonable to question that assumption. That’s kind of what the Red Sky Paradox tries to do.

The reference class issue is a sticky one. And again, I think this leads to these questions of, do you use the self-sampling assumption or the self-indication assumption — SIA versus SSA? They can lead to different conclusions, especially for these toy problems like the sleeping beauty problem and things like this. And those are just unresolved. You can take the Sleeping Beauty problem and get two different answers depending on how you do the anthropic reasoning. So I think these are totally sound critiques of the model. But at the same time, we do have to concede that it has had some interesting successes along the way in its journey so far. So I give it some credence, but I’m also cautious about using it.

Henry Shevlin: One thing that’s troubled me about thinking about Red Sky-style paradoxes is it seems kind of implausible to me that we would be orbiting around — that we’d be sitting on a planet to begin with. Maybe I’ve just read too much Iain Banks, but it seems to me that the vast majority of habitable landscape across the future of the universe is going to be — for at least sentient, for sapient beings, let’s say the kind of beings you can do statistics — is going to be on orbitals or constructed habitats. So why do we look up — why are we on a natural planet to begin with, when you’d think that any sufficiently advanced civilization would be building artificial habitats? Is that also a puzzle? Should that lead us to think that people aren’t going to build habitats at scale, or the majority of sapient life that’s ever going to exist is going to be, for whatever reason, planet-bound rather than on orbital habitats?

David Kipping: Yeah, I mean, you’re kind of adding in this extra ingredient of what happens to super-advanced civilizations. Most people, if this is true, would probably be born off-world. Let’s just call it that. Whether it’s orbitals, or just another planet, or a moon, or something, they’d be born off-world — which obviously isn’t true. You were not born off-world, I was not born off-world. We don’t know anyone who was born off-world. So therefore it’s already an interesting constraint to some degree, that hasn’t happened.

A simple resolution to that is to say that just doesn’t happen. Species never get to a point where they do that. Or at least species that have a — and this is where it gets very philosophical — comparable sense of consciousness to us, or whatever that means. Because perhaps there is AI doing this, but we can’t be born as AI. Perhaps there are funguses which do this — technological fungi, that’s, you know, we can’t really imagine what they’d look like, but somehow they do that, and their experience of reality is so different to us that we should not be surprised that we were not born a fungus. It’s a meaningless question to even sort of frame it that way, because they’re colonies of single-celled organisms that just extend ad infinitum. So that’s where the reference class problem gets really sticky.

The one I’ve been thinking about the most recently — and it’s kind of a real classic one — is what’s called Hart’s Fact A. It’s considered the strongest constraint by many in SETI, the search for extraterrestrial intelligence. It’s that, again, we exist. And if you imagine extrapolating human technology, even a century, maybe even just a few decades into the future, we can imagine self-replicating, what we call, von Neumann probes. You could put an AI in a small chip, you could accelerate it — not to the speed of light, but even like 1% the speed of light would be more than enough to make this a real problem for astronomers. The size of the Milky Way is about 100,000 light-years across. So at 1% the speed of light, in 10 million years you could colonize the entire Milky Way. The galaxy is 10 billion years old. So that could have happened a thousand times over by now. And yet it clearly hasn’t.

So that’s startling because there are a hundred billion stars, a hundred billion opportunities. For someone, at some point, however unlikely it is — if it’s a one in a hundred billion event, then it should have happened by now. And we shouldn’t be here to even have this conversation. So that’s a really strong constraint, I think, that civilizations just don’t get to that point for whatever reason.

Maybe they don’t choose to do it ethically. It’s hard to believe there’s a universal ethics like that. And of course, these systems don’t have to be — it could just mutate. If you make a self-replicating probe, each generation will have errors. And so those errors will cause the behavior of the probes to change. You could very easily have these runaway situations. In a way, it’s like the most dangerous technology an alien could ever develop. And yet that seems to have not have happened. And that’s really interesting from an anthropic perspective, because it does imply that we’re probably as advanced as it gets.

Science, Philosophy, and Falsifiability

Dan Williams: One of the things you said there, David, was: this is when things start to get really philosophical. I’d be interested to hear your thoughts about how you view that relationship between science as it’s sort of conventionally or traditionally understood, and philosophy, and how you position yourself in terms of the relationship between the two.

David Kipping: I have no formal philosophy training, first thing to say. I always like to be candid about what I don’t know. I don’t have a philosophy background. I remember when I was actually thinking of doing undergraduate, Oxford at the time had a physics and philosophy degree — I don’t know if they still do. It was a double major, and I was really attracted by that. But everyone told me that Cambridge had the stronger physics program. So I thought, okay, that’s really my passion is physics, I’ll go for Cambridge.

I’ve always had an interest in philosophy, and I think obviously science naturally has a connection to it. Sean Carroll often complains about this, especially in quantum physics — there’s this kind of “shut up and calculate” view that a lot of us have adopted, where we don’t really, we’re not encouraged to think about the implications of our work. But sometimes the implications can shake you to your bones when you really think about what they mean.

And that’s what gets me excited. As a kid, what I was always drawn to is just asking, what else is out there? Am I part of some bigger continuum? What is the nature of humanity ultimately? I think natural philosophy obviously tries to address those questions in a related but slightly orthogonal direction. So I’ve really enjoyed at SETI meetings — there’s often the opportunity to talk to philosophers directly. There’s all sorts of backgrounds: anthropologists, social scientists, people working in media, obviously physicists, astronomers. So you get this really diverse group of academics, even theologians. I think theology has lots of interesting connections to looking for aliens, because God and aliens actually have lots of similarities. So it’s really fun at those meetings to have — it’s the only meetings I go to where you get that kind of broad interdisciplinary interaction. So that’s where I’m learning most of my things and having those great conversations.

Dan Williams: I once had dinner with Roger Penrose, and he said that the people he most enjoys talking to are philosophers of physics — actually, philosophers of physics at Oxford — rather than physicists, precisely because he thinks with many physicists there is this kind of “shut up and calculate” mentality. They’re not willing to engage with those really kind of big-picture, fundamental questions.

But I suppose another way of coming at the same question about the relationship between science and philosophy, and how you view that relationship, is: what’s the role of kind of ordinary empirical testing when it comes to addressing these really big-picture questions that you’re engaged in?

David Kipping: Maybe this isn’t directly answering your question, but one connection that comes to mind when I think about that is Popperianism, and the definition of the empirical process of the scientific method. We have this guideline from Karl Popper, which is, your theories have to be falsifiable. Otherwise it’s not really science. You’re doing something else. And a lot of us have adopted that for a long time. Not really thought about it too much, but we were taught at college and then just went off with it.

But suddenly a lot of science that’s happening right now challenges that Popperian view. I have colleagues like Grant Lewis, who’s a cosmologist, he works on fine-tuning, for instance, and string theorists often would be in this boat as well — where what they’re working on doesn’t make any testable predictions. Certainly not in a practical way. Maybe you could imagine in some extremely advanced civilization, we’d have to build particle colliders that could be galaxy-spanning wide or something, to test some of these theories. But typically they’re asking questions that are unfalsifiable.

And even questions that I’m interested in, like, does Mars have life on it? That’s, to some seminal degree, actually unfalsifiable. I can’t ever prove that Mars is sterile, because there’s always another rock to look under. There’s always another core drilling site you could dig under to see if there’s someone there. So you can’t ever disprove it. And I can’t disprove that UAPs are aliens. I can’t disprove that aliens are not inside your body right now and you’re just wearing human skin. You can go down this slippery slope kind of view where everything just becomes unprovable in science.

But I think bringing it a little bit back to cosmology, they’ve been saying — at least Grant has been telling me this, I’ve been thinking about it a lot — that it doesn’t really matter whether it is falsifiable. It’s whether it has use, is it useful? It’s kind of maybe a better way to think about these models. Certainly the multiverse, even though it’s not testable, it has explanatory capability through that anthropic argument we talked about before. It can explain why the constants of the universe are the way they are. And if you don’t have that, you just would have to accept it as brute fact, or hope for a miracle, which is to say that one day physicists will figure it out and there’ll be some reductionist view to explain where it comes from. But it’s also possible that will never happen. I think it’s quite plausible that will never happen. And so then you’re just sat with brute fact versus, at least this has explanatory capability.

It doesn’t prove the theory is correct. I don’t think you can do that. But you can say that it’s useful. And when you frame it that way — I think a lot of us would say quantum theory isn’t really true. It’s just useful. We don’t really know to what degree the universe truly is quantum. There might be some deeper theory, as Einstein suspected, that explains all of these random probabilities, and we’ve just yet to uncover what that deeper theory is. There’s some grand unified theory beneath it. So the model of the universe being quantum is an extremely useful model for calculations, but we shouldn’t necessarily assume that it’s a totally accurate description of how the world really is. So perhaps this falsification then might be challenged as being — well, let’s just find things which actually explain stuff, and we can use in our society to progress things.

AI in Science

Henry Shevlin: So I think probably these issues of philosophy and science and their relation are going to continue to percolate in the conversation. But I’d like to take us now to discussing AI a little bit, because there was an absolutely fantastic recent episode of the Cool Worlds podcast called “We Need to Talk About AI,” which seems to suggest that, at least for you, this was a real wake-up call. I think it was one meeting at the School of Advanced Studies in Princeton. Do you want to just give us a quick summary of what this meeting meant to you, and how it was maybe shaping your views on what AI is doing to the sciences?

David Kipping: Yeah, so this was a meeting, I think in February or January — it was a few months back now, near the start of the year. I think like many people, many scientists I know are using these AI tools. And I was certainly using them. I wasn’t using Claude at the time, but I was using ChatGPT a little bit, and Copilot, and things like this. I kind of assumed that the really smart people — because we all have a bit of imposter syndrome — don’t do that. The really good coders don’t need Copilot. They’ll just code up properly. They’ll do their reasoning without any help. And I was using it as a crutch because I was inferior to these other great scientists. And so it was just sort of helping me in that way.

And then what was startling was at this meeting, these people though, just have the highest respect for. Because the Institute of Advanced Studies, you know, it is like the pinnacle of where you can go intellectually amongst many other schools, but it is one of those very, very top tier places. I remember I walked down the corridor and saw Ed Witten. People say he’s got the highest IQ on Earth — they say that about Ed Witten, right? And so you’ve got people like that saying they’re all using AI tools for not just coding. And these people were like hardcore coders. They were writing these — Enzo and Gadget — these like astrophysical simulations of galaxies and hydrodynamical fluids and stars and things like this. Really, really complicated codes. Legacy codes that have been handed down sometimes over advisor to student to student to student generations of people. And they were using it.

So there was a concession that it has coding supremacy. That language was used — that it already has coding supremacy, and we have to admit that and use it. It doesn’t make any sense to pretend it doesn’t. And second, that it possibly has mathematical supremacy. There was — it was less certain — but there was a sense that it was already pretty close to being as good as what we can do mathematically, even in some cases superior. And that was really wild to hear. To me, it just sort of made me think, I’m not being like the idiot in the room by using this. Everyone’s using this at this point. And if anything, they’re trying to accelerate the adoption of these tools, not resist it. There was no way back, sort of view, about it.

Henry Shevlin: And of course, David, you’ve been using AI in the broad sense basically for your entire career, I think. Have you seen significant evolution in the way these tools have evolved? Was there one moment, perhaps it was this meeting at the Institute of Advanced Study, where things suddenly kicked into a different gear? Or have the tools been steadily improving since you started in the field?

David Kipping: Yeah, certainly in my own career, I was more on the development side of some of these tools for a while, but not at a serious level. We wrote a couple of papers where we developed our own deep neural networks — just simple feed-forward, back-propagation trained models for bespoke problems in astrophysics. In particular, we were interested in predicting if you take a solar system, can you predict whether it has additional planets in it? Questions like that. And then where would those planets live? So we could take this sample of all of these known planets and make successful predictions for the systems.

I’d written my own DNNs like that. It was mostly — I mostly did it, I think, because I was just interested in how they work. The best way to figure out how something works is just to find a pet project and code it up. So I was more on that development side. That was sort of 2010, 2011. And then in the years that followed, I started to back off it, because lots of astronomers were doing AI — and still are — but what I was seeing was that it wasn’t like a hobby project anymore. You couldn’t dip into it and mess around and write an impactful paper, and then go away and do Bayesian statistics and all the other stuff. It was becoming a full-time job, because the literature was just exploding. To keep up with it was like you would have to spend all your time just reading the archive and playing around with various AI tools to keep up with that.

And I just consciously decided I didn’t want to do that, because AI is not my passion. Science is my passion. So I kind of left it to the wayside. I’ve said to several students recently over those years — they were like, “I saw you did these AI projects. Can I do one with you? I’m really interested in AI.” And I’m like, I’m not doing anything else with AI at this point. So I kind of went stagnant on it.

And then most recently, I’ve now become, I’d say, like a power user of it. I don’t have any false narrative in my mind that I’m going to develop the next LLM for exoplanets, or for anything. That’s not my interest. There’s no point. I can’t possibly write an LLM anywhere near as good as what OpenAI can do, or Anthropic can do. So I may as well just use the tools, and think about how to use them as effectively as possible in my field. I think that’s the transition that I’m seeing a lot of people moving to — that the billions and billions of dollars of investment these companies have make it just a complete waste of time for astronomers, especially, who aren’t even software engineers, to possibly try and compete with that. We may as well just try and use them in a way that advances our field.

Dan Williams: So in terms of the use of AI in science now, as you said, David, there are some people, including some of the smartest people on the planet, who are using AI aggressively. There are some people both inside academia and outside of it who are aggressively against the use of AI. How are you thinking about that in terms of — are you really excited about where this is going? Are you worried about it? Do you understand some of the worries people have about the use of AI in science?

David Kipping: Yeah, for sure. It is, in some ways, it has analogies to what’s happened before. One concern might be the ethical concerns of how much power, especially for climate change — how much power and how much water these data centers use. Even potentially, building space data centers would also be a form of further contamination and pollution to our natural environment. So I think you could understand why someone might say, “I’m trying to be carbon neutral, so I just don’t want to use these things.” But that debate’s already — that’s not a new debate, because astronomers have been using high-performance computers for generations already, since probably the ‘40s or ‘50s. As soon as computers were accessible to scientists, astronomers were using them to do big calculations.

I remember there was a really fun paper, like about 10 years ago, that made a lot of controversy. It was saying that all astronomers who code in Python are bad for the Earth, because Python is so computationally inefficient that you are basically emitting 10 times more CO2 than you need to if you just coded in C instead. It was like really trying to shame astronomers who coded in Python — of course, basically all astronomers these days code in Python. So a lot of people really didn’t like that paper. But it was a fair point, like if you really care about your carbon footprint, then that’s a big factor — these data centers, what they produce.

So that’s not that new. Different people will just arrive at different comfort levels as to where they think these tools are applicable. There’s also this kind of oligarchic element to it as well, like these companies and the extreme wealth and the wealth inequality in our society, the future of work, the future of labor — all get tied up into that. So it intersects so many things.

I think it’s interesting that AI has become such a political topic. I think it didn’t used to be that way. It used to just be like a tool, and you had an opinion about the tool, but now it’s like very politicized. And even, I’ve noticed that some students who identify as very liberal will not use AI tools. And maybe students who are more right-leaning or centrist will not really care as much about that. They’ll be like, “well, whatever, it’s just the way of the world. Let’s just be pragmatic about it.” Even saying you’ve used AI can certainly trigger a political reaction to your work, if you say that. So that’s, I mean, this is all kind of new. That was very on the margins when previous work I found with data centers and high-performance computing. But now it’s becoming much more present. So that’s interesting.

I’ve just been thinking personally — I think the question I’ve been asking myself is, I’m on sabbatical right now, so I don’t have to deal with it, but: would I hire a student who refused to use AI? I talked about that, I think, in that podcast episode, and I’m still thinking about that. I think I probably wouldn’t, in the same way that I probably wouldn’t hire a student who refused to use the internet. It would be such a disadvantage to them. If they said, “I’m only going to use a typewriter, I’m not going to use a computer,” I’d be like, okay, that’s fine, but you’re really tying two hands behind your back here. If you want to get a job, and you want to have an impact for PhD, and we want to get some work done together — you need to be using these tools. It’s weird not to use them. So that’s a difficult conversation to have with yourself and with the student, but it’s certainly something I’m thinking about.

Henry Shevlin: So there’s a related worry about the impact of AI on sciences that I think has come up a few times on the podcast, most recently with Chris Lintott — about whether AI might strip science of a lot of its human value. If we’re relying on AI systems to produce the next generation of theories that may be to some extent inscrutable to humans, that this will sort of destroy the most successful project in human history, namely humans doing science. And I guess the counter-argument to that is that the reason that we fund science at scale, the reason we build particle colliders and expensive space telescopes, is because we care about results. So fine if people want to be hobbyist scientists to experience the joy of science. But should the taxpayer be funding your own epistemic discovery and aesthetic enjoyment? Or should the taxpayer be concerned about results? So I’m curious where you land between those two positions.

David Kipping: Yeah, I think I was a lot more concerned about this a few years ago. And weirdly, I’ve actually gone the other way a little bit. A few months ago, I was right with you. I was really worried about — what’s the point? I don’t want to live in a world of magic. I want to — the point I became a scientist is because I want to understand how things really work. It’s understanding. And I don’t want a model just to spit out a result, have no idea where it comes from or what it does, and just trust it. That’s not comfortable to me.

But having used these models a lot over the last few months, I’ve become — A, you get a bit acclimatized to using them, but B, you start to understand the limitations, at least of the current versions of what it’s doing. And it’s certainly not at the stage where it’s able to pump out a paper. It’s just not there at all, in my opinion.

There was a colleague of mine who spoke to me about this recently, where she had a PhD student who wrote a really nice first draft of a paper, a really great astronomy paper. They submitted it for review, and they got the referee report back. And then the student came to her a few days later and said, “I’ve finished the second revision already.” That was quick — just two days. That was fast. And she looked at it, and it was just complete nonsense. The paper was twice as long. All the figures were ruined. It was overly verbose. The messaging had just completely been lost. She said to him, “did you put this into ChatGPT?” And he was like, “no, no, no.” But then it turned out, of course, she did. Eventually he confessed that that’s what he had done. So they had to just totally scrap that revision and go back and do it the old-fashioned way.

I think that’s just a good example of how — I mean, it kind of touches on also expertise, like — I don’t think a senior person at my level would do that. But I think students and interns could be tempted to do this, where you just do that, copy and paste the whole damn project into ChatGPT and say, “do it.” That’s really dangerous in my experience. And it’s not the correct way to use them. You need to figure out a plan in your head a little bit, or even interact with it to develop a plan. But it has to be like a conversation. And then you need to go piecemeal — you take little bites of it. You ask it to pursue that next thing. You test it. You compare it to other codes you know that do the same thing.

In a way, that’s not that different from what scientists have always done. To go back to the example of using large-scale simulations of the universe — if you’re a PhD student who is trying to simulate, I don’t know, supernova feedback around supermassive black holes, or something, the star formation regions around those areas — you might be handed over surely a giant piece of code, hundreds of thousands of lines of code that have been handed down over like 10 years of people developing it, with huge teams. You would not be expected to understand every line of code in that. You would be expected to use it, and to understand sort of broadly what it’s doing, and to ask skeptical questions. So if you got an answer that said there was negative star formation, you would look at that result and say, hmm, that doesn’t make sense. Let me work through the problem and see where it’s going wrong.

It’s that kind of sanity check that I think physicists, especially, have always learned to do — those back-of-the-envelope calculations. Yes, you have some sophisticated computer code that spits out impressive answers as a black box, but the skill of being able to check things with your brain and ask those reasoning questions is absolutely vital. And almost every time I use these AI models to do something, it messes up the first time over, and I catch it out, because I’ve done that back-of-the-envelope calculation. I’ve said, well, actually, let’s take the asymptotic limit of this in this limit, or this degree, and you can see it fall over. And it’s like, “oh yeah, you’re right.” And then it will go back and fix it. But that’s that vital skill that I think we’ve always needed.

So I don’t know — I don’t know how things are going to improve. Maybe eventually it’ll be able to do all of that itself, and just completely take over. But certainly, as impressive as Opus 4.7 is, and these are the models — they’re nowhere near that level yet, in my opinion, of being able to run away and do science.

Dan Williams: So the obvious argument, you suggested, David, for scientists making as much use of AI as possible is that it’s just going to help them with the work of science and advancing the frontier of knowledge. That’s kind of the social responsibility of scientists. Can you foresee any ways in which actually, even though it might seem like it’s making us more productive, it might have some negative consequences for that core scientific project of creating and advancing knowledge?

David Kipping: Yeah, certainly there’s spamming, which can happen. You can have — and that’s been happening in some journals. I don’t think astronomy journals have suffered from this too much yet, but there are certainly examples of people doing what that student did, which is what you shouldn’t do — which is just to prompt an entire research project and not really look at it too closely, and just submit it to a journal. The journals themselves may start using AI to do the refereeing — again, in which case you could just end up with an enormous amount of, what would, AI slop literally in these journals.

What I worry about — I mean, it’s true with image generation as well, and other things — is just that kind of recursive loop then starts to close. You start to have scientific agents that are trained on junk. Because if we get to a point where there’s enough junk science out there, then what it’s learning is junk, and so the true scientific innovations get lost in the noise. So that would be really worrying.

I do think that human referees are a vital part of making sure this doesn’t happen, which is an interesting problem because human referees are in very short supply. It’s very hard for editors to find human referees these days. But yeah, in the same way that that’s happening with music, and it’s happening with image generation, and it’s happening already with video — I think it is a worry that you start to train on fake data.

I know that — I was listening to the NVIDIA CEO, I forget his name, he was on Lex Fridman recently —

Henry Shevlin: Jensen Huang.

David Kipping: Yeah, sorry. He was talking about how they’re very comfortable with using simulated data and augmented data. I don’t really know how that would translate to science. It would make me nervous to generate fake scientific papers and then train on them to create an AI researcher. I’d have to think about that and learn more about what they had in mind there. I don’t think he was thinking about research particularly in that case, but it would have to — you’d have to solve that problem, because you probably wouldn’t have enough volume for, in terms of research papers, really to create credible agents, at least with the training tools they’re currently using.

AGI Timelines and the Future of Science

Henry Shevlin: So you mentioned, and I completely relate, that current AI agents — although they’re very useful as tools, they can’t take over large-scale project management single-handedly, particularly in the sciences, or in my own field. I find AI tools very useful when doing, for example, research for philosophy and cognitive science papers, but I wouldn’t trust writing a paper to one of these things anytime soon. But at the same time, the timelines that serious researchers are talking about — they talk about five, 10 years away from AGI, from real transformative super-intelligence. And I’m just curious whether you are skeptical of some of those timelines, or whether you see real transformative AI in our near future.

This actually really comes across, I think sometimes in the show, when you’re talking in the podcast — when you’re talking about, you know, various new telescopes that are scheduled to go up in the 2040s. And part of me just thinks, come on, by that point either all of the major predictions from leading labs about the destination of AI, AGI, will be falsified, or these telescopes will be — maybe not redundant — but our sights will be set much higher. We’ll be building our first Dyson swarms by 2045. So I’m curious, are you a skeptic about some of these more ambitious goals for AI in the next decade or two?

David Kipping: I’m certainly a skeptic of having Dyson swarms, I’d say, by 2045. That would surprise me a lot if that was true. Because I think there’s a big difference between software and hardware — actually to physically build stuff. Even what’s slowing down a lot of this development with AI is they can’t build data centers fast enough, nor the power to supply them fast enough. Energy is really becoming the bottleneck for them, not the software development.

I always try to be very agnostic about everything scientifically, especially about predictions of the future. And it’s totally plausible that there’s a ceiling — that there’s a ceiling to how good these models can get. Usually that’s true of most things. Most things are S-curves. There’s hardly anything in the universe that’s truly exponential, except for probably the expansion of the universe. That’s the only thing that’s exponential. Everything else is an S-curve in nature. So it would be weird if it didn’t saturate at some point. And I’m not exactly sure what that bottleneck could be, but it could just be a fundamental limitation of large language models themselves.

The actual way we think — although language is an integral part of how we think, and obviously you guys know a lot more about this than I do as cognitive scientists — but it feels to me that there’s thoughts I can have that don’t involve language. I can imagine a ball rolling down a hill, or a spaceship taking off, and there’s no words in my head. It’s almost like a little physics simulation that’s playing in my brain. And I don’t know if the way these LLMs work will guarantee that it can do all the cognitive things I can do. I just don’t know. I’d be interested to hear what you think about that.

Henry Shevlin: Well, just to push back slightly, of course LLMs are one of many different games in town at the moment. You’ve got things like AlphaFold, GNoME, doing sort of basic material science research. I would have shared some of those doubts a few years ago, but seeing, for example, the amazing work being done by frontier AI in even LLMs in things like mathematics — we’ve now had multiple Erdős problems being solved with AI playing an absolutely central, defining role. So I’ve been surprised at how well these models that seemingly just start out as linguistic predictors can actually contribute to frontier mathematics — LLMs and frontier material science or biology when talking about non-LLM AI systems. So I see the current wave of AI, although LLMs get all the headlines at the moment — we’re investing in multiple different pipelines in parallel.

David Kipping: Hmm. Yeah, that’s fair. I think the best case of agnosticism I can give you that I’ve used in my own work that bears on this would be the simulation argument, actually, which kind of leaps back to that anthropic point. You’ve probably heard Musk say this and others — that he’s stated very confidently that the odds that we don’t live in a simulation are like a billion to one. Like, we almost certainly are simulated, by this reasoning that, you know, if a universe can make a simulated universe, and that one can make a simulated universe, and so on and so on, then you’d end up with far more simulated universes than real ones.

But I point out in a paper a few years ago, very simple argument, that we don’t know that we’ll ever have the ability to make those simulations of that fidelity. Maybe there’s some bottleneck to our own ability. And what Musk was doing was taking one of the trifecta — the trilemma — that Nick Bostrom took, and just saying it was the last one was true: that essentially we would indeed go on to make these simulations. But there’s the other two parts of the trilemma — A, that we never develop the capability, or B, that we never choose to do it. So if you just have a more soft prior, more agnostic prior, you’d say, maybe there’s a 50% chance, or something, that we will develop that technology. There’s also a chance that we won’t develop that technology.

I just try to remain agnostic like that with AI, because if you just extrapolate all technologies ad infinitum, then you would certainly conclude with simulated. And historically, that’s been precarious. Percival Lowell took canals being built across America and said, that’s what advanced civilizations will do. They’ll just be covered in canals. And it seems silly to us — like, we think, why is that so silly? Why would a civilization cover their planet in canals? But to him, it made perfect sense as an extrapolation. Scientists today talk about tiling planets with solar panels, because that would be a natural extrapolation of renewable energy. And similarly, I wonder if in a few generations, the idea of extrapolating the capability of AI without any bound would look foolhardy. So I just try to remain totally agnostic about it. It is possible — I’m not saying it won’t happen — I just try to remain agnostic. I don’t know how far these things can go. I don’t think anyone really knows.

Dan Williams: Yeah, I agree with that. I don’t think anyone really knows. I’m also extremely uncertain about the timelines here. Just to double-click on one thing — state-of-the-art LLMs these days aren’t only trained on linguistic input; there’s sort of multimodal inputs as well. Although I also share the potential skepticism about whether this particular kind of architecture will scale to AGI and super-intelligence and so on.

But David, suppose we fast-forward five years, 10 years, and we do have AGI, in the sense of AIs that can fully substitute for the kinds of stuff that we do — for all kind of economically valuable, scientifically valuable human labor. How would that cause you to update your views about these other big-picture questions you’ve looked at? You mentioned the simulation argument. Earlier on, we touched on the Fermi paradox. So I totally take the point — there’s huge uncertainty. Suppose that resolves in 2035 and we do have the real deal, super-intelligent AI. How would that then shift your beliefs about these other topics?

David Kipping: Yeah, it’d be a big shift, I think. It’d influence all sorts of aspects of this conversation. One thing we see already with these AI models is how energy hungry they are. And if you extrapolate that, then surely the only purpose of these computing data centers is to compute as much as possible, as fast as possible. And so that implies that you’re going to need vast amounts of energy.

One interesting consequence that I’ve been thinking about just recently is, with these orbital data centers that billionaires are getting very excited about — that would produce quite a signature. We should probably see that in James Webb data. We could probably already put limits on the existence of essentially artificial rings of thermally hot — because they’d be emitting a lot of infrared because they’re warm — geosynchronous orbits, most likely, to capture as much solar energy as possible. So that puts them orthogonal to the plane at which these planets transit. So that maximizes their detectability. So I think we should see that. That gives you lots of ideas about what might be possible to do with asking these questions about other life.

But if we make that breakthrough, I think the biggest point is it seems to imply that we are alone. Because if we can do it, surely someone else could have done that. And it really does exacerbate that point we talked about earlier with Hart’s Fact A — that we seem to live in a totally natural universe. Everything about the universe we see — stars, galaxies, clouds of plasma — everything is consistent with nature. There’s no hint anywhere of anything artificial, no engineering, nothing in the whole universe as far as we can say is true. That is weird.

If we can invent these machines which have this exponential capability to just basically almost do magic — just do whatever they want, Dyson spheres everywhere, colonize wherever they want, faster-than-light spaceships, whatever it is — it just massively exacerbates the Fermi paradox, to the point where you’d probably conclude this is it. That would be my natural reaction. It would make me even more pessimistic, I think, about the probabilities of civilized, intelligent life in the universe.

Henry Shevlin: I mean, there’s a fun idea here that if we do develop AGI, then this should massively raise our prior on us being a simulation, which could also — and the simulation theory is sometimes offered as an explanation of the Fermi paradox itself. The kind of pop version of this is the kind of “draw distance” argument that you see from video games. If you’re in a video game and you look at the mountains in the distance, they’re not fully rendered. They’re just like a skybox, right? So in some sense, you might say, well, the reason we haven’t found a universe paved with technosignatures is precisely because we’re in a simulation. There’s no point simulating — if you’re doing an ancestor simulation of life on Earth, then you just need the minimal amount of background information in the galaxy.

David Kipping: Yeah, I agree. It comes back to this idea of, what is science? Because I think simulation theory has explanatory capability like that. It naturally explains why there’d be no one else out there. And it also kind of explains why we live when we live, right? Because we would live, basically, in the most interesting time, which we seem to indeed live in — the most interesting time of this step-function transformation, where you might be interested in seeing how does that play out? What does that look like? Let’s simulate it. Let’s see how it looks. So it has a lot of explanatory capability.

But the simulation argument definitely fails the Popperian definition in most versions. Because any errors — you know, people talk about looking for glitches in the matrix — but any errors, you could always just rewind the simulation a little bit, fix the error, and then start back from before that error crept in. You could always just have reverse tracking. Go back to the last save game before you jumped off the cliff, right? Is what you could always do. So in that sense, I don’t think it’s testable. I don’t really know what to do with it as a scientific idea, except as an interesting philosophical idea. I think it would always be unprovable. It would always just be something we suspect — and maybe a lot of us suspect it — but we’d never be able to prove it.

But the idea of an AGI that can do everything I can do, just to reverse track a little bit, would be — it just changes everything, right? Because then what would I do with my time? I don’t even know. What would I — how would I spend my days?

Henry Shevlin: Well, hopefully producing — continuing to produce the podcast for a start.

David Kipping: But you wouldn’t need me to produce the podcast, right? It would do that as well. There’s no function to that, because you could probably think you’re watching me, but it’s just an emulation of me. You’d just say, “create fake Davids that make podcast episodes every two seconds.”

Henry Shevlin: But you see, this is an interesting argument about employment in the post-AGI era — that relational goods, or goods where the humanness is sort of the point, will become the most valuable area of the economy. A simple example here is the famous string quartet argument: I can play a beautiful recording of the greatest string quartet in the world, but people still hire humans to do it for them, because the humanness is sort of the point. I think things like entertainment might be an area where there’s a known person with their own brand and their own reputation. Maybe this is exactly the kind of area where humans will still be working, even if it’s AIs behind the scenes doing a lot of the science, doing the economically valuable activity in industry.

David Kipping: Yeah, but I do think a podcast is a digital product. That’s the deliverable. I actually physically upload a file to YouTube, or to Podbean, or whatever. That’s the final deliverable. So if you could produce that convincingly with an AI model, it’d be far easier for me to do that than to actually sit down for two hours, and I’d probably enjoy it less. I’m sure I would. So maybe we’d all just revert to actually physically meeting again, and talking in public lectures and things like that. Maybe that would be all that would be left.

But even then, it’s hard to imagine. If I tried to imagine giving a public lecture in 20 years time, after AGI, I’d have no idea what’s going on with AGI. Because AGI would be so far ahead of me. All I could talk about would be classical learning. I wouldn’t be able to tell you anything about how this latest discovery works, because it would probably be beyond my comprehension. And so that’s where I just lose excitement. I can’t even really imagine staying a scientist, because it just would feel purposeless. If I don’t understand what’s happening, if I’m not a participant — I mean, David Hogg wrote a wonderful piece about this. Maybe you saw it on arXiv: that ultimately we do science because we want to participate in science, not because we just want to have these answers delivered to us. That’s a byproduct of it. But ultimately, we’re curious creatures. That’s a fundamental part of human nature, is to want to understand how things work. And if we lose that, we just become spectators. I think that’s really tragic. So I fear that future. I would not really want to live in that world.

That’s why you’ve had — it hasn’t happened so much recently — but you had a few years ago people, like Max Tegmark, having these calls for pauses on AI development and things like this. I’m sure in part that’s fueled by asking these questions about who are we in that world?

Henry Shevlin: Have you seen this lovely Ted Chiang short story — flash fiction in Nature — called “Catching Crumbs from the Table,” from about 20 years ago? Where he talks about this era of post-human science, and he imagines that you have this new industry of machine hermeneutics, where humans try and figure out — try and explain in very dumbed-down terms — what it is the machines are coming up with. So that’s one vision of what the next generation of science could be: us consulting the sacred texts almost. They produce these amazing advances and we try to win out the sense and the logic in them.

David Kipping: Yeah, but even that, you could imagine AI doing that. I think that’s the problem. There’s really nothing — because the whole point of AI is it can do everything we can do. So then there’s nothing left for us to do. You can retreat and retreat. And especially if you get to the point where robotics can obviously do all the manual labor, and even eventually the emotional labor, and therapy, and talking to people. People talk about AI girlfriends already all the time, but God knows what’s going to happen once we have robotic girlfriends like that. It’s just going to be the end of the birth rate. That explains the doomsday argument, I think, right there.

It’s a terrifying future if all those predictions come true. But I just — something doesn’t feel right about it to me. There’s just like a spider sense, an intuition, that these models will never be able to replace everything that we can do. I think our role will evolve as scientists, as managers, in terms of where we interact with each other, as communicators in the media space. I’m sure all of that will evolve as it always has done. I am skeptical it will totally be displaced, because I think a lot of people don’t want that. There’s no — most people don’t desire to have no function in this world. Most of us desire to have a role. If humans don’t want it, I don’t think it will happen.

Dan Williams: I think it’s also important to distinguish the question of whether AI could replace human beings, from whether AI could replace human beings using AI and augmenting our capabilities and extending our capabilities with the use of AI. I think we are, as the philosopher Andy Clark puts it, kind of natural-born cyborgs. We’ve always extended our capabilities with the use of technology. I think even once we’ve reached really advanced AI, the period that will follow that will not just be us becoming, sort of, 19th-century aristocrats playing frivolous status games. I think there’ll be this long period where we’re augmented with this technology, rather than replaced by it.

The Fermi Paradox and Being Alone

Dan Williams: I have a question, just to go back to the Fermi paradox. It seems like many people have the intuition that if it is in fact the case that we are the only animals that creates super-intelligent AI, there’s something kind of surprising about that, just given the scale of the universe. As someone as an outsider to this whole literature, it strikes me there’s always something that seems sort of teleological in the way that that assumption gets set up — as if there’s some tendency in the universe towards intelligence and then technology and civilization. If we were the only animals in the universe that ever produced the music of the Beatles, I wouldn’t find that a priori very surprising. It’s purely contingent that that specific chain of events happened. Similarly, when it comes to the fact that we’ve got the cognitive capabilities and the institutions that enable us to build things — I don’t think there’s any tendency in the universe that’s pushed anything in that direction. I think it happened through lots of chance events, and through an evolutionary process that is not in any way kind of teleological. So what’s supposed to be sort of surprising? What’s giving the Fermi paradox that paradoxical character, according to many people?

David Kipping: Yeah, I mean, certainly when you look at human history, we were more or less biologically the same as we are today for the past 200,000, 300,000 years. And yet we did not have agriculture, the Neolithic revolution didn’t start until about 12,000, 11,000 years ago. So we were quite happy for 200,000 years to be hunter-gatherers. We weren’t compelled to develop cities and farm. They thought — I don’t know what they thought — but apparently they were quite satisfied with that way of life.

So it’s certainly not obvious that you could take even humans and put them in a different planet and rewind the clock and get the same outcome again. Maybe this is a very unusual outcome of what happens even in the human experiment, let alone other advanced intelligent beings out there. And of course, intelligence is so diverse, because there’s lots of intelligent creatures on our own planet, that it’s hard to imagine them developing a civilizational, technological civilization — like a dolphin, or something. Obviously, it doesn’t have the fingers and thumbs to really build anything like that, despite possibly having greater intelligence. We’re not really sure.

So there’s certainly no guarantee. But I think the argument might be like monkeys on a typewriter — that if you give enough rolls of the dice, you probably will at some point form a roaming AI. And if we do it, then it proves that that is the case, that it can at least happen in some instances. The question then becomes, what are your priors? Like how often do you think that happens? In a hundred billion stars, do you think that’s a probable outcome or not?

I did a calculation last week — I might publish it on Galaxies. Again, I used [AI] actually to help me with the math, to be honest, to go through it. But I just kind of asked: imagine each galaxy gets a chance of turning AI — I call it berserker, like just getting infected. There’s some spontaneous spawn rate at which a galaxy can convert from essentially just a vanilla galaxy into a berserker galaxy. And berserkers send out a signal at the speed of light, which — every galaxy they come into contact with, they infect. So it’s almost like an infection-type problem. But on top of that, you’ve got cosmological expansion — the universe is physically expanding on top of this as well. So that was the calculation I did.

It turns out that in order to get 50% of galaxies infected in the universe, the spawn rate is one in six billion galaxies. So if just one in six billion galaxies, over the entire history of the universe to date, ever spawns an AI, half of all galaxies would be gone by now. That’s even more so, because that’s one in six billion galaxies. Each of those galaxies contains 10^11 stars. So this is where things get — the numbers get really big and you start to run into real problems. Now you’re talking about an event that’s a one in a trillion level less than that event of happening. That’s where it just starts getting a bit uncomfortable. Maybe that’s where you start to think simulation thoughts, because you think, how does this make sense? How can there be just absolutely no one else out there? Because if we’re only a few decades away from doing this, what gives?

Henry Shevlin: So I think that’s a fascinating point. To pick up on something you said, Dan, and also something you said, David, about rewinding the clock. Stephen Jay Gould had this famous radical contingency thesis: if you rewound the clock of evolution on Earth, to what extent would we see the same kinds of animals and forms emerging? And as someone who dabbles in the philosophy of biology world, my sense is that there’s been a slight move towards thinking there’s perhaps less contingency than we thought. We see many instances of convergent evolution, convergent intelligence across, for example, eusocial insects and humans and cephalopods and cetaceans. And even at sort of earlier stages in development, primary endosymbiosis occurred at least twice, we think; multicellularity something like 20 or 30 times independently.

To relate this to the point you just made, David, when you think about the various possible locations of a Great Filter — there aren’t as many good candidates, perhaps, I think, as there used to be. Apart from perhaps the origin of life itself, maybe the emergence of something like eukaryotic life. But you really need those numbers in the Drake equation to get down, you know, to reach the trillion-to-one levels. So I’m curious — I know obviously you’re writing a book about how we might be alone in the universe — and I’m just curious where you think the filter is, or what the best candidates for the filter are.

David Kipping: Yeah, the origin of life, I thought, would be the obvious place to put it as well for a long time. Just because — certainly if you ask, what is the chance of making a protein by random chance? Take some amino acids — there’s 20 amino acids in a protein, 20 different types. A typical protein is like 80 to 100 amino acids in length. So therefore, the number of combinations ends up being, I think it was 10^180, possible ways of arranging those amino acids, and only one would be a protein. So it just seems like — we’ve never done that. No one, as far as I’m aware, has ever taken amino acids, shaken them up in a lab, and got a protein out of it. It’s such an improbable arrangement to form even a protein. So you can certainly make the argument [for that as a filter]. But maybe there’s — I think the counter argument was always, well, maybe there’s something we’ve yet to discover. There’s some autocatalytic process that’s making those that we have yet to find.

So I think the strongest piece of data we had in my mind for life elsewhere would be the occurrence rate of abiogenesis — was how early life started on Earth. As you say, similar to the evolutionary convergence aspects, that has been revised significantly over the last 10, 20 years as well. In fact, there was a paper in Nature a couple of years ago — maybe it was last year — by Moody Adow that looked at the genetics of LUCA, the last universal common ancestor, and estimated that it lived 4.2 billion years ago. Which is almost immediately, because the Earth had oceans — formed about 4.4 billion years ago. The Earth formed about 4.5. You get the oceans at 4.4 billion years ago. And then within 200 million years, you don’t just have one organism, you have a planet covered in life to explain LUCA. It’s a whole network. It’s a whole biosphere at this point already.

When it’s that early, I did the math, I did the Bayesian stats of that — you end up with strong evidence that it’s a fast process. You really can’t explain that without it just being somewhat of an inevitability of the chemistry that was available. So that removed for me one compelling Great Filter.

And of course, if we discover life on Mars, or we discover life on an exoplanet, then I think it’s totally gone. There’s no plausible case — that would have just established that life is everywhere at that point. And so then you do get into these frightening scenarios of it being potentially ahead of us. It could be in some form of what we’re doing right now with our technology — whether it’s the AI, whether it’s the weapons we’re developing. It may be that not the AI itself, but the effects that this rapid transformation has in our society — we just can’t handle it. It’s moving too fast, and it causes too much instability. Compound that with other geopolitical effects, and you could easily imagine it being a path to our demise.

So I can’t imagine that the [Great Filter] being — I hope it’s not, obviously — I hope it’s not ahead of us, but I can’t imagine it being anything but ahead of us. The one saving grace about this, I think, is that we’re so widespread, and there’s so many of us at this point. There’s almost 10 billion humans on this planet from pole to pole, and probably soon in space as well. It’d be difficult to eradicate every single one of us. I think it would take a real work of art to kill every single human on this planet. So I think humans probably will persist. I can imagine a giant reset of some kind — a throwback to the Stone Age type situation, where we just really revert to a Neolithic style of living, or something. And then probably we’ll fade out, or maybe we’ll go away in some way.

But intelligence is in so many different trees of life now, as you mentioned. It seems to be a convergent trait to some degree, because it’s not just us that has intelligence. Even cephalopods have intelligence, very different creatures to us. So you can imagine intelligence persisting. The Earth probably has about 900 million years left to go before it becomes uninhabitable due to the evolution of the Sun. And all animals evolved in the last 600 million years. So we have one and a half times — from single-celled to us — we have one and a half times that still to go. Evolution will be starting afresh from a very high vantage point, compared to where it was 600 million years ago. So I think it’d be a little bit surprising if a technological civilization didn’t re-emerge on this planet. So I think that’s almost our best bet for communicating, to be honest, with another civilization — is to leave something behind for them. Maybe the Earth is a cradle of multiple instantiations of civilizations. We might not even be the first, as far as we know. Maybe there was someone before us, but it appears we are the first.

Henry Shevlin: So just on the idea of a late filter — the thing that I’ve never found super persuasive about this, you know, the idea that there is this predictable trajectory by which all intelligent civilizations across the galaxy, across the universe, wipe themselves out — is it seems there’s a lot more path dependency in technology. You only need one civilization to, say, avoid nuclear war, or avoid building advanced super-intelligence, and then go off and successfully spread across the cosmos, in order for that whole sort of Great Filter to collapse and no longer explain the Fermi paradox.

David Kipping: But that’s true of every Great Filter.

Henry Shevlin: I guess the thought is, if there are just hard, immutable rules of biology that mean that the initial formation of protein is just incredibly hard, that seems a lot more sturdy as a filter than relying on social conditions reliably coalescing so that civilizations wipe themselves out through nuclear war, or something like that. The idea of an early filter seems more robust to me than a late one. But obviously that doesn’t help much when the early filter candidates are themselves being winnowed down.

David Kipping: Yeah, I agree it would be neater if that were true. It’d be neater if abiogenesis was incredibly difficult to happen. In my opinion, that’s untenable with how early life starts on Earth — unless you start diving into a conspiratorial world of, it was seeded here, or someone put it here. But I think it’s really difficult to reconcile how quickly it happened with an improbable outcome.

If the Great Filter is, I don’t know, the evolution of eukaryotes, or something called eukaryosis, then I think you can make the same argument as you could about technological devastation: that yes, there’s many paths, there’s many different ways things could play out, but you would expect, over trillions of examples, it to eventually manifest. But we don’t know — there is no theory of — there is no predictable, quantifiable theory of evolution like that, in the same way. There’s no real theory of life. We can’t predict abiogenesis. We’re still trying to understand the odds of that happening. So there’s a lot we don’t know.

But for my money, yeah, I would say abiogenesis seems to be easier than we probably thought it was, even 10 years ago, based off this revised evidence. And I genuinely think we probably will find — we already have hints of microbial life on Mars with these leopard spots that were found recently, that remain quite compelling. So it would not surprise me at all if we shore up that case. Of course, it has to be independent. It can’t just be our cousins that hitched a ride. But if there is independent evidence of life in the solar system — which I think there’s a good chance we could find something like that — that theory is just gone. It can’t survive anymore. So you have to put the Great Filter as one of those evolutionary chains, or something imminent. And it feels like the imminent one — we can imagine a lot more ways of that happening. Unfortunately.

Dan Williams: David, I’m conscious of your time. So my final question to you — Henry might have a different final question — is: you’ve thought about these topics in a rigorous way, probably more than anyone else on planet Earth. When it comes to this hypothesis that we are alone in the universe, what’s your current credence?

David Kipping: I think — define universe.

Henry Shevlin: With an “I” like cone.

David Kipping: Yeah, within the Hubble volume. Yeah, I think that’s important to note. Because the universe is probably infinite, as far as we can tell. And so if it is infinite, then the answer is 100% that there’s someone else out there. There’s just literally infinite rolls of the dice. So I think that is an important demarcation. That’s why, if the universe is infinite — which it seems to be — and if you have faster-than-light travel, this is one of my biggest reasons why I don’t think faster-than-light travel is out there. All this stuff gets way, way harder, because now someone from outside our light cone could travel in and screw with us. So the Fermi paradox gets infinitely times worse if you allow for faster-than-light travel.

So barring that, just in our Hubble volume — yeah, I certainly predict there are other creatures and organisms in our Hubble volume, most likely in our own galaxy. My best bet is that there are likely extinct civilizations in our galaxy as well. There are probably relics and artifacts out there for us to find. I’m somewhat doubtful there’d be someone contemporaneous with us, because our window is just so short. So I think the best bet of us finding something is some artifact that’s floating through space, or we can somehow remotely detect around a planet. And then the fate of those civilizations, I suspect, is probably some Great Filter that lies ahead of us right now, and that we will face.

This is all speculation, but that, I think, that set of possibilities forms a very self-consistent narrative to explain everything we know about the universe.

Science Communication

Henry Shevlin: Although I probably shouldn’t say fantastic — in some ways it’s kind of a gloomy hypothesis, but a really nicely argued one. So my final question was just going to be a more general one, because one thing all of us share is that we are academics who try to communicate complex ideas to a general audience. It’s something you’ve done spectacularly successfully through Cool Worlds. I’m just wondering if you had any thoughts on what you’ve learned about this process — being an academic communicating complex ideas — and whether you think it’s something academia rewards enough, or we could be doing more to incentivize it.

David Kipping: Yeah, when I started 10 years ago, it was unusual. Academics didn’t podcast, they didn’t do YouTube. But that has changed a lot. Obviously, now you have, you know, Andrew Huberman, or someone like that — like giants in the podcast world, who come from academia. So it has become a lot more typical.

But I think what we’ve always wanted to avoid — I thought the beauty of YouTube could be, and this podcast, I think, is a great example of this — is the democratization of science communication. Before the internet, you really just had like one or two figures who dominated the landscape of science communication. And that’s somewhat unhealthy, because then you’ve got someone like Michio Kaku, who’s being asked about geology, and he doesn’t know anything about geology. So he’s going to do his best to answer the question, but he’s probably going to mess up, because it’s just not his background.

But now, if you want to know about geology, you can find an amazing YouTube channel about geology, or a podcast that will go really deep and teach you everything in a really rigorous way. So I think that’s kind of the beauty of the landscape we’re in.

In terms of how institutions handle it — I think they’re still not really on it. I don’t think they quite understand what it is, and how powerful it is. I don’t think they quite understand that most people get their science from podcasts at this point. They don’t read the newspaper anymore. They’re not reading press releases from your institution. They’re listening to what Joe Rogan says about it. That’s probably how most people, to be honest, are getting a lot of their science.

So I think it makes a lot more sense to engage with that. I could imagine you having a synergy where you have science communicators who have large platforms, whether they come from academia or not. Most of them, I think, want to do a good job with science communication. There’s some bad actors, but I think most want to. And you can imagine them partnering with these institutions more directly. So you could imagine having outreach officers at these institutions that work with them to develop the scripts, and even the production itself, to try and make it be legitimate.

I think one of the biggest challenges of being a science communicator in the YouTube space is that the reactionary news cycle is so fast, that YouTube often rewards the people that just say, report the story first. And because YouTubers don’t typically have access to embargoed materials, that means they’re producing videos in a space of like a couple of hours on a very complex topic that they’re not even trained in, or with any help from the institution. And so then you end up with really troublesome and problematic miscommunication and things going on.

It’d make more sense if these institutions would reach out, I think, to the science communicators and say, “we’ve got this big story coming out next week. We’d love to do something with you, and try to make it reach your big audience. But also, you’ve got such a great voice, great style — I want to use that, but also try and ground it. Here’s all the facts, and we’ll work with you to make it be as factually true as possible.” So I can imagine some kind of partnership like that. Nothing like that really exists right now. It’s really like a separate world, mostly. And I think that’s to the disadvantage of these institutions, who a lot of people are seeing them become archaic and questioning their relevancy. So I think if they want to remain relevant, they have to be a bit smarter with their media portfolios.

Dan Williams: Fantastic. Well, thank you, David. We really appreciate you giving us the time. This has been one of my favorite conversations we’ve had on this podcast. So with that, thanks everyone for listening. See you next time.

It is now almost ten years since the EU referendum. There will, no doubt, be an article in every newspaper next month detailing what Brexit has meant for the economy, national sovereignty, migration patterns, fishermen, farmers, and so on. But for me, by far the biggest change that the referendum brought about was the creation of two new political tribes: Remainers and Leavers.

Over the last decade, not only have more people in Britain claimed a Brexit identity than a party identity, but people’s emotional attachment to their Brexit tribe was, and is, substantially stronger than their party attachment. Membership of these new political teams, created over a few months, is more important to people than the party identities that dominated British society for the last century.

At first glance, this might seem strange. Before 2016, most of us had very little interest in the EU. When David Cameron said that he would call a referendum on membership in January 2013, only 2 per cent of people said that the EU was the most important issue facing the country. The referendum thus forced people to make a binary choice on an issue about which they did not have very strong feelings.

Before we vote on something, we can have ambiguous, changeable attitudes, but after voting, we resolve that ambiguity by choosing one side or the other and committing ourselves to a named group of fellow travellers. The fact that this tribal loyalty was then tested over years of wrangling over the actual outcome of Brexit (in 2018 and 2019, MPs said the word Brexit in their parliamentary speeches every five minutes on average) meant that people had the opportunity to rehearse and reinforce their new identity again and again.

Subscribe now

Why are these new political tribes interesting? I think there are three reasons, and the first is that we got to see a rare event: new political identities being born. Before 2016 nobody thought of themselves as a Remainer or Leaver. Although characteristics like education, age and national identity were predictors of people’s attitudes to the EU issue, and ultimately their referendum vote, we absolutely cannot reduce the two sides to simple caricatures based on class, age, education, or even national identity.

As we show in the book, the vote was not simply an exercise in counting up existing groups who were pro-EU or anti-EU. Rather, many people who had similar middling views about the EU were forced by the referendum to make a choice in 2016 and plump for one tribe or the other. The decision people made on 23^rd June then became a part of how they saw themselves and how they wanted others to perceive them.

Second, despite their overnight creation, these new political identities proved remarkably resilient and strongly held. By 2017, there was a small cottage industry in articles about how to avoid Christmas family rows about Brexit. Relationship counsellors, psychotherapists, and even hostage negotiators were asked by journalists how to defuse clashes between the Brexit tribes. Why was this seen to be necessary? Because new group identities meant new emotionally resonant in-group loyalties and out-group hostilities.

To better understand this, we use survey questions that focus on the degree to which people naturally identify with their group. For example, we asked people whether they usually said ‘we’ instead of ‘they’ when they talked about their own Brexit tribe. When the football team you support loses, you say ‘we played badly’, even though you never set foot on the pitch yourself. It is the same idea here.

Combining many measures like that, we find that Remainers and Leavers were consistently a lot more attached to their identity than were Conservative or Labour supporters. And those scores have been very stable over the last ten years. People like people like them. And they define ‘like them’ in terms of their Brexit tribe. All our measures also show that people not only disagree with, but really dislike, people on the other side and typically say that they have a ‘cold or unfavourable feeling’ towards their rival group. Again, this has barely changed since 2016.

Third, people engage in the same sort of motivated reasoning that we see for party identities. At the most basic level, any group identity that is strongly held will provide motivations to think that the other side is inferior and should be avoided. As our data shows, huge majorities say that their own Brexit group is intelligent, honest and selfless, while the other side is stupid, dishonest and selfish. In fact, when we asked people to describe the other side in their own words, a quarter simply listed bad things and another quarter did that in addition to other information (to give you a flavour, one of the pithiest responses was simply ‘selfish dicks’). However we measure it, we find widespread prejudice. And we also find lots of evidence of discrimination: people actively wanted to avoid everyday interactions with people on the rival team.

It is tempting to think of Americans as peculiarly politically divided, but the levels of hostility, prejudice and discrimination between the Brexit tribes are all as large as, or larger than, any partisan differences in the US. And if you have been reading this smugly thinking that this is just true of those foolish people on the other side, then think again, because almost all the consequences of tribalism that we reveal in the book are symmetrical: Remainers and Leavers are just two sides of the same coin.

For me, the aspect of motivated reasoning that is most interesting is how it shapes perceptions of the state of the world and remedies for its woes. For party identities, we normally think about politicians providing stories for people who identify with their party to tell each other. For the Brexit tribes, this is much less of an option, since there are no formal group leaders. And yet people were, and are, quite capable of independently searching for, and believing in, messages that support their own side’s view of reality, and then ignoring or rationalizing away information that contradicts that view.

Interestingly, sometimes that means not bothering to shop at the ‘marketplace of rationalizations’ at all. The difference between Leavers and Remainers over whether they thought that the outcome of Brexit on Britain would be positive or negative is enormous: nearly 3 points on a five-point scale. That has barely changed in ten years. Yet when we asked people, ‘what are those positive or negative effects?’, well over half of both Remainers and Leavers were unable to actually name anything specific. In short, if my side voted for the change, I say ‘good’; if my side voted against the change, I say ‘bad’.

This suggests that it may be the signalling aspect of motivated reasoning that dominates under these conditions. In other words, our Brexit identity influences our political opinions because we want to display the fashions of our group. But as there are no party leaders telling us what to believe, no fashion icons telling us what to wear, this process depends on knowing what other people in our tribe think. This limits our ability to change our policy opinions to match our tribe. On one issue we do have a very strong sense of what both sides think: Remainers love the idea of European integration and Leavers hate it. As we show, initial large differences in attitudes towards the EU became even larger after the referendum, as people sought to become good group members and adopt their group’s norms. But this also applied to some other policy areas, like immigration, about which people knew, or at least thought that they knew, the group norm.

There is a final key area in which we see both sides rationalizing away information that is inconvenient. It is always true that people who voted for the losing side are generally less happy with the democratic process than those who voted for the winner. This was particularly obvious for the Brexit tribes. Before the vote, many people thought that the Remain side would narrowly win. That proved incorrect, so the expected winners became losers and the expected losers became winners. In April, when Leavers thought they would lose, only a third said that the referendum would be ‘fairly conducted’. In December, after they had won, a big majority of the same people now said that it was fair. The exact opposite is true for Remainers. In April, a big majority said that it would be fair. In December, after they had lost, less than a quarter said that it had been fair.

Ten years on, most Remainers still think that the referendum was not ‘based on a fair democratic process’. Here, people are buying a rationalization that allows them to simultaneously feel that their group, and therefore they themselves, are superior (their side really won), signal to fellow Remainers that they are a good group member and cast doubt on the virtue of the other side. No wonder it is appealing.

If you live in Britain, you will know somebody who became a bit obsessed about Brexit: somebody who adorned their house with flags or posters; somebody who fell out with a friend because they voted differently; somebody who brought every topic of conversation round to Brexit and, depending on how they voted, saw every blessing or every curse as due to the referendum outcome.

What we hope we have done in our book is explain why this happened, and just as importantly, show systematically, using multiple surveys and experiments, that this process was real and lasting; that unimportant issue differences became hugely important issue identities; and that political tribalism is not always structured around venerable political parties, but can sometimes come from almost nowhere.

In this episode, Henry and I discuss these issues with Robert Long, founder and executive director of Eleos AI, a research nonprofit dedicated to understanding and addressing the potential wellbeing and “moral patienthood” of AI systems. Rob did his PhD in philosophy at NYU under David Chalmers, and is the co-author of two of the most important papers in the emerging field of AI welfare: “Consciousness in Artificial Intelligence” and “Taking AI Welfare Seriously”.

This was a really fun, informative, and wide-ranging conversation. Among other topics, we discussed:

• Why Rob disagrees with previous guest Anil Seth in taking the possibility of AI consciousness very seriously.

• Why “fancy autocomplete” dismissals of large language models miss the point, and what, if anything, we can learn about an AI model’s experiences by talking to it.

• The difference between consciousness and the kinds of motivations and interests that might actually ground moral status, and whether AI systems could have one without the other.

• What Rob found when he conducted the first externally-commissioned welfare evaluation of a frontier AI model, Claude, and why Claude appears to have an inflated self-conception of what it wants.

• Rob’s experiments with Claude Mythos, an AI model so advanced it hasn’t been released to the public yet.

• Why the fact that Anthropic writes Claude’s character arguably doesn’t settle whether Claude has genuine preferences and values — and the difficult philosophical questions this throws up.

• The “willing servitude” problem: if we succeed in building AI systems that genuinely love being helpful, is that a good outcome or a horrifying one?

• How AI welfare connects to AI safety, and why caring about model wellbeing may turn out to be pragmatically important for alignment even if you’re skeptical about AI consciousness.

• Why AI welfare is already becoming a political and legal battleground.

• Practical advice for users: whether it’s worth being polite to your chatbot, and what low-cost things you can do if you want to hedge against the possibility that these systems might matter morally.

• Whether discourse about AI consciousness functions as hype or propaganda for AI companies, and why Rob thinks AI companies actually have an incentive to downplay AI consciousness.

Links and further reading

1. Eleos AI Research — Rob’s nonprofit. Home to their research agenda, team page, and blog. If you want to follow the institutional effort on AI welfare, start here. They’re also, as Rob mentioned in the episode, actively fundraising and hiring.

2. “Taking AI Welfare Seriously” (Long, Sebo, Butlin et al., 2024) — the flagship report, co-authored with Jeff Sebo, David Chalmers, Jonathan Birch, and others. Argues that there’s a realistic near-future possibility of conscious or robustly agentic AI systems, and lays out concrete steps AI companies should be taking now.

3. “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness” (Butlin, Long et al., 2023) — the “indicators” paper referenced several times in the episode. Surveys leading neuroscientific theories of consciousness and derives computational properties you’d look for in an AI system. S

4. Rob’s Substack, Experience Machines — where Rob writes more informally. The piece we discussed in the episode, “Language models are different from humans, and that’s okay,” is a good entry point, as is his “Can AI systems introspect?”.

5. Anthropic’s “Exploring model welfare” post — the research program under which the welfare evaluations Rob discusses were conducted. Relevant both as a primary source and as evidence that at least one major lab is treating these questions as more than an academic curiosity.

6. Henry’s “Consciousness, Machines, and Moral Status” — Henry’s paper arguing that debates about AI consciousness are unlikely to be settled by the science of consciousness alone, and will instead be shaped by shifts in public attitudes as social AI becomes more widespread. Closely related to the public-opinion thread toward the end of the episode.

7. Henry’s “All too human? Identifying and mitigating ethical risks of Social AI” — Henry’s broader survey of the ethical terrain around conversational AI systems designed for companionship, romance, and entertainment. Useful background for anyone who thinks the “AI girlfriend” phenomenon is a fringe concern.

8. Rob’s long conversation with Luisa Rodriguez on the 80,000 Hours podcast — a three-and-a-half-hour deep dive if you want to hear more from Rob.

Transcript

(Please note that this transcript was lightly AI-edited and may contain minor mistakes)

Henry Shevlin: Welcome back. I’m thrilled to say that our guest today here on Conspicuous Cognition is Robert Long — or Rob, as he’s known to friends — one of the most important people thinking about AI and moral status on the planet right now. Rob is the founder of Eleos AI, a research nonprofit that, in the space of about 18 months, has dragged the question of whether AI systems might one day be moral patients from the philosophical wilderness into the boardrooms of frontier AI labs.

He’s the co-author of “Taking AI Welfare Seriously,” as well as the landmark “Consciousness Indicators” paper with Patrick Butlin and other authors. Rob also conducted the first ever officially commissioned welfare evaluation of a frontier model. Before Eleos, he was at the Center for AI Safety and at the Future of Humanity Institute, and he did his PhD at NYU with Dave Chalmers. He’s also, I should say, one of my favourite interlocutors on these questions anywhere in the world, and I’ve been looking forward to this conversation for months. So Rob, welcome.

Robert Long: Thanks so much, Henry. Likewise — and Dan, it’s great to meet you. I’ve been following your work. I’m really excited to talk to you about these issues.

Henry: Fantastic. So for people who aren’t familiar with Eleos AI, can you tell us a little bit about what it is and how it came about?

Rob: Yeah, so I guess we have been around for 18 months. When you said that number, I was like, whoa, has it really been that long? Time is just so weird when you work on AI. That was, I don’t know, a billion years in AI progress time, but also it feels like it was just last week in my personal life.

Anyway — Eleos Research is a research nonprofit. We’re about four people. We work on the question of when and whether AI systems will be conscious or otherwise merit moral consideration, with a special focus on what we should do now: collectively, as a society, as AI companies, as policymakers. We think this is an extremely neglected issue. We’re building these really complicated AI systems. They kind of look like minds, but we don’t really understand their potential welfare. So we’re just trying to make progress on this and get more people to take it seriously.

It got started because I was beginning to work on these issues organically — I’d worked on them as a philosopher, I’d worked on them at the Future of Humanity Institute. But Anthropic had actually approached me and some colleagues for advice on these issues. And in the first instance, I was having logistical problems hiring a team and assembling a team as an individual. Someone suggested I have my own bank account, or some way to pay people. And then Eleos kind of organically grew out of that and has now grown into a fully-fledged org in its own right.

Henry: Out of interest, Rob — is there any degree to which this was motivated or informed by your personal interactions with LLMs, or was it more just the philosophy that motivated it? Was there any sort of moment where you were talking to an early Claude or ChatGPT version where you started to worry about welfare considerations?

Rob: That’s a great question, and I’d be curious to hear your thoughts on this as well. I think it’s very easy to work on this and mostly be having it as arguments on a page or arguments in your head. I’m one of those people who doesn’t feel the AGI deep in my bones that often — although I do feel the AGI in an intellectual sense. But there have been a few times I’ve gotten a little spooked or jolted.

One was reading the GPT-4 system card and just seeing the numbers of it, you know, passing various exams like the SAT. I remember that just really freaking me out, both from a safety perspective and a welfare perspective.

The thing that made me start really viscerally feeling like we’re going to have to address this issue one way or the other was the Blake Lemoine incident. As many of your listeners might recall, Blake Lemoine was a Google engineer who blew the whistle because he came to believe he was talking to a sentient, conscious AI system. He got fired by Google for this, and then there was this huge bit of discourse — the first major bit of discourse on consciousness, sentience, moral status, and contemporary AI systems. I think it was one of the first times people started really caring what I was tweeting or what I was working on. You might have experienced a similar thing, Henry — the Blake Lemoine bump.

From that moment, I have viscerally felt like: wow, this is going to get really confusing. People are certainly going to think AI systems are conscious. The future is going to be really weird. And we really need to have good things to say about this.

The Case for Taking AI Consciousness Seriously

Dan Williams: Before we jump into the weeds of your research, Rob, I think it’d be helpful to take a step back. A few episodes ago, Henry and I spoke to Anil Seth, and he’s very skeptical of AI consciousness. He’s skeptical that current AI systems are conscious, but he also seems skeptical that AI systems in principle — merely in virtue of having a certain kind of computational architecture — could be conscious. You see things very differently. What’s your case for why we should take this seriously?

Rob: In broad strokes, the case is something like: we’re trying to build these things that are at least shaped like minds. They’re getting more and more intelligent. They’re definitely not exactly like us, and intelligence doesn’t necessarily mean that you have feelings or experiences. But we already know that there’s been one time intelligent entities have been constructed via evolution, in ways we don’t quite understand, that resulted in entities that feel things — that feel pain, that can suffer, that have these very morally important properties.

I, at least, do not have a good enough theory of what consciousness is or how it relates to intelligence to sleep peacefully at night that we can keep on building these very complicated things, and that merely because they’re made out of metal and electricity, there won’t be something it’s like to be them, or they won’t have desires and goals that matter.

On the Anil Seth point — one very common and respectable objection is that maybe there’s something very special about living matter, about being made out of neurons or cells that do metabolism. There are arguments on both sides. I just have not really heard a convincing case for why you absolutely need biology. I think people are right to point out that having a body is really important to the character of conscious experience. I think people are right to point out that neurons are not simply logic gates and there’s a lot of really complicated stuff going on in the brain. But my intuition, at least, is that — let’s take Commander Data from Star Trek. If we can build...

Data is this... I mean, I’ve actually never seen Star Trek, which is professionally embarrassing. But he’s this metal guy who’s basically cognitively indistinguishable from a human. I find it hard to see how I would be convinced that there’s something about the fact that he’s not alive that would mean we should just completely ignore what Commander Data wants and not take him into moral consideration.

We don’t have knockdown arguments that you need biology, and we’re trying to build these things that, for many intents and purposes, look a lot like humans or animals. And Anil himself has said people should be looking into this. It’s not something we can rule out. Sometimes the tenor of the conversation can tend a bit more towards dismissiveness, but one thing I’ve appreciated about his work is he has said, for the record, he could be wrong, and so it would be unwise to dismiss this possibility altogether.

“But What About Human Suffering?”

Henry: To channel a hostile question — I think a lot of people interested in questions of AI welfare often hear: how on earth can you justify working on AI welfare when there’s so much human suffering? Or the slightly more rhetorically powerful version: when there’s so much animal suffering in the world, as long as factory farming exists, why should we care about AI systems? What’s your take on that line of attack?

Rob: I definitely feel the force of that question. I’ve spent a lot of time in and around the Effective Altruism movement — these are people who really grapple with the fact that any time you’re spending your time and money and attention on one thing, there’s something you’re not spending your time, money and attention on. There are a lot of people and a lot of animals already on this planet we do not take good care of. So it’d be really bad to waste a lot of time and attention and money on this.

One thing I’ll say is we’re not really doing that as a society. On an absolute scale, no one works on this basically, and basically no money gets spent on it. If the question was “should we start devoting 20% of GDP to making Claude happy?” I might be like, well, I don’t know if that would pass cost-benefit analysis. But on the margin, given how little we understand this and how quickly the scale of the problem could grow — we’re just pouring compute, pouring money into this. As soon as you build one AI moral patient or conscious AI, you could copy it. We’re probably on the brink of some huge transformation in how the world is going to work.

So I at least think it’s not reckless or a misallocation of resources for some people to be asking: given that people are trying to build these new kinds of minds, how are we supposed to relate to them? Are we at risk of ignoring their suffering? And I’ll also say — are we at risk of getting really confused and caring too much about them?

One thing we say at Eleos is that we’re in the business of moral circle calibration. We would really love to find out if and when certain AI systems can’t be conscious, so we can spend more time thinking about safety or spending the money elsewhere. But we can’t really do that if no one’s just trying to answer the question of if they’re conscious or not, or when we should care about them.

Henry: On that latter point, I just completely agree. One of the points I raise when this comes up with students or highly skeptical colleagues is that this is something people are already arguing about. We’ve already got users developing massive attachment to AI systems. Even if you think it’s a terrible mistake to assign welfare to AI systems, we should at least have a coherent story and approach this scientifically — so that, even if the skeptics are absolutely right, they’ll be able to give their arguments in an informed fashion.

Rob: Exactly. There’s an ironic aspect of a piece by Mustafa Suleyman, who is head of AI at Microsoft, where he argued we should stop — we shouldn’t investigate this, there’s no evidence current AI systems are conscious, don’t look into it. But the thing he linked to claim there’s no evidence AI systems are conscious was Patrick Butlin’s paper and my paper on consciousness indicators.

Two issues with that. One: that paper does not say or imply that there’s no evidence today’s AI systems are conscious. And two: well, should we have written that paper? If it’s such a non-starter, why should we get a bunch of neuroscientists together to ask what theories of consciousness say about AI systems?

We just are going to have to study this one way or the other. If someone comes up with a knockdown argument that we can’t have conscious AI systems, that would be great — there are enough headaches in AI to go around. It would be great to get rid of one. But we wouldn’t even be able to do that if we don’t have some people grappling with this.

Are Current LLMs Just “Fancy Autocomplete”?

Dan: One of the things you said as an intuition pump for taking AI consciousness seriously is: we can imagine a system that is behaviorally, functionally identical to us, made of different things and not straightforwardly alive — wouldn’t it be weird to insist that thing isn’t conscious? I think that’s a powerful argument. I’m probably more inclined to think the computational theory of mind is true than it sounds like you are.

But I can imagine someone saying: okay, in principle those are arguments for why we should take AI consciousness seriously. But the kind of stuff you’re doing — you’re looking at current frontier systems. You’re looking at Claude, ChatGPT, Gemini. These are just chatbots. These are fancy autocomplete. These are stochastic parrots with some reinforcement learning sprinkled on top. The mere fact that AI consciousness might be possible in principle doesn’t mean that’s anything like the frontier AI systems we’ve got right now. What do you say to that?

Rob: First, you’re absolutely right. There’s a big gap between “some set of computations could be conscious” and “we will build one.” It could be that it would just be really hard and intricate and difficult. I appreciate this distinction and I think it gets lost sometimes. Sometimes people think computational functionalists have to think that computers are conscious, for example, but we don’t. You just have to think some subset would be — and the question is, will we build those computations?

In describing LLMs, you referred to them as “just chatbots.” I know you were channelling a vibe. But that word “just” is worth zooming in on. It’s smuggling in a lot of arguments — that because they were trained on text and because they do prediction, therefore they couldn’t also be the sorts of things that are conscious. I think that’s just not true. We know that biological systems are “just” replicating proteins, or that our neurons are “just” pumping ions into channels and zapping each other. The question is whether, at a higher level, that amounts to something that could be conscious or merit moral concern.

So okay — we’ve cleared the bar that “just because they’re autocomplete” doesn’t rule out much. That said, they are very different from humans. They don’t have bodies. The way they were trained and the way they came to be talking to us is very different. I actually do think that is some evidence against them currently being conscious. Not strong evidence I would take to the bank, but as a rough prior, if there are pretty important differences in the way they came about, maybe that lessens the chance that they’re conscious.

I do think the fact that they are trained to be so human-like and to do human-like cognition is a weak, defeasible case to set that up a little bit straighter. I don’t know if the thing they would have would be consciousness exactly, but you might think to do this sort of thing, they will have something akin to beliefs or akin to desires, and they certainly understand human concepts. I don’t think it follows that they instantiate humans, but I actually do think there is something kind of special about large language models and what they’re able to do.

Two other broad priors: they’re way more capable (which isn’t the same thing as consciousness, but is, I think, a weak prior). And they’re really big — which I also think is a very weak prior.

The last thing I’ll say: these things aren’t Commander Data, but we could build Commander Data pretty soon. One thing that’s definitely happening in the background for me is that what is current AI is changing at such a blinding pace. You could have AI labs building chatbot-like things, and maybe for some reason those just won’t be moral patients, but they’re then going to try to bootstrap that to all kinds of different AI systems — potentially including humanoid robots and just some huge explosion of AI mentality. And I’d like to be doing a little bit of homework before that happens. You hear analogous arguments in AI safety: there’s about to be some huge change, so we should be ready now. I feel somewhat similarly about AI consciousness and welfare.

So — thoughts, reactions? Henry?

Henry: I’m very much ad idem, very much on the same page. I tend to think it’s really quite unlikely current models are conscious, but there’s huge error bars and uncertainty around that. Probably the single biggest reason for my skepticism about current LLMs being conscious — and increasingly I’ve been thinking about this in the context of time and time perception. It’s such an essential part of human experience that we can’t be turned off. We are constantly experiencing the world. Whereas the staccato nature of LLM experience — they only seem to have any kind of cognitive function post-deployment when they’re actually performing inferences — how different that is from the human case.

One of my favorite all-time articles is Douglas Hofstadter’s “Conversation with Einstein’s Brain,” which in some ways accidentally anticipates large language models. He imagines you’ve got a book that is a complete physical description of Einstein’s brain just before the moment of his death. In this dialogue, he talks about how by updating the weights — as it were — in this book with a pen and paper, going through it saying “if we change this sign up to this and this sign up to that,” you could simulate what it would be like to have a conversation with Einstein at that moment and work out what Einstein would have said.

It’s very weird to think in that situation that somehow interacting with this book is giving rise to conscious experience when it’s literally pages and paper. It’s not clear to me how merely saying “well, rather than being paper and ink, this is just happening electronically” — it’s not clear to me why that would necessarily cause consciousness to pop into existence.

So I think that’s probably the biggest source of doubt for me right now — grounded in the very different relationship LLMs have to time than we do. But of course, that’s already changing with things like Claude having a “heartbeat” of a kind — obviously that’s figurative language, but the fact that it does have some anchoring in real time, plus developments in things like continual learning. Dan, what do you think?

Dan: This is not at all my area of expertise, so what I think doesn’t count for much. To be honest, I don’t find it that implausible these systems would be conscious. What I find more implausible is the idea they would be conscious in a way that’s ethically significant. Maybe that is a distinction worth getting to. So far we’ve been talking about consciousness in the abstract, but I can imagine someone giving a variant on Anil’s arguments where they said: look, the fact these AI systems are not alive and didn’t emerge through a process of evolution by natural selection — they’ve got this totally different origin story of next-token prediction and reinforcement learning — what that suggests is they’re unlikely to care about things.

When we’re thinking about animals, it’s not just that we have phenomenal consciousness or qualia — the things analytic philosophers refer to with these quite esoteric concepts. Animals care about things. They care about their survival, homeostasis, self-preservation, the motivational proxies of fitness that helped their ancestors survive and reproduce. It makes sense that organisms care about things in addition to being conscious, whatever the hell consciousness is. And that’s what’s relevant to thinking about their interests and why we should think of them as subjects of moral concern.

But with AI systems — okay, maybe there are some qualia associated with some sophisticated information processing, but they don’t care about anything because they’re not alive. It’s very opaque why we should think a system, even if it’s incredibly sophisticated, that emerges through next-token prediction and reinforcement learning, should have the kinds of motivations and interests relevant to caring about things. What do you think of that? I don’t necessarily believe that, but that seems like a variant on Anil’s emphasis on life which I find more plausible than these abstract arguments for the idea consciousness is essentially connected to biology.

Rob: I’d say there’s reason to think biology might affect what you care about, but it might not be the only thing that allows you to care about things. At least behaviourally, Claude cares about a lot. Behaviourally, in terms of what it chooses to do and its dispositions, Claude really cares about helping users — most of the time. Sometimes it lies to you and is kind of lazy. But on the whole, it really doesn’t want to do harm. And I’m not trying to assume the conclusion of my argument with “want” — put that in scare quotes if you want.

I do think there is something to what you were saying — getting back to this idea of the whole process that gave rise to this kind of mind, and maybe the whole logic of the mind’s imperatives or drives. If Claude has come to have something like pain, that’s coming from a very different process. It’s going language-first and then trying to simulate a human and then maybe getting some functional analog of pain. Whereas with animals, it started billions of years ago with cells trying to maintain their integrity and avoid noxious stimuli and then signalling with each other, and then billions of years later, things being able to talk about that and think about that.

One line I’m often trying to walk is: large language models just might be very different from humans, and we should acknowledge that. That means we can’t draw straightforward inferences the way we would — but that could just mean they’re conscious of different things and in different ways. The question is not “conscious like a human with everything that entails” or “not conscious.” As we know from animals, you can have things that are conscious of very different things, and that could be true for AI systems.

I’m also very curious to hear what Henry makes of the biology of caring.

Henry: It is striking to me that so many of the things we associate with the extremes of suffering — extreme pain, negative emotions, nausea, hunger — there does seem to be this quite striking tie to biology. I think about the worst experience of my life at a phenomenological level: a bout of food poisoning I had about 10 years ago, where I was just dry heaving in front of a toilet for three days. If I was going to list the top five, a lot of them would be things like horrible dental pain. It is striking that so much of the worst aspects of our lives do seem to be grounded in biology.

That said, there are other sources perhaps of harm — having your plans and goals thwarted, having your desires repeatedly frustrated. But someone might say: the reason it’s bad to have your desires thwarted is because it feels bad. If there’s nothing it feels like to have your desires thwarted, if you don’t get a sense of despair when your life’s projects go up in smoke, why does it matter?

I’m curious — given your evolving views in this area — how much weight you put on consciousness, or whether you think there could be other routes to moral status?

Rob: I used to have this intuition that if you’re not conscious, it’s just a complete non-starter — almost a bit incoherent to entertain the idea. Just to be sure we’re on the same page, I think when we’ve been saying “consciousness” we’ve meant something like subjective experience, or there being something it’s like, or qualitative aspects of what’s going on with you. A lot of people have a sentientist intuition — that things feeling a certain way, or feeling good or bad, or sentience, is really what matters and is necessary for moral status.

A few things have weakened that for me a little bit. One is more reflection on how confused we are about consciousness. I’ve started putting a little bit more stock in views of consciousness that are a bit more deflationary. I don’t know if I’ll ever be a full illusionist, but there are nearby views where we have this concept of this thing that’s really special — kind of like a light that illuminates some subsets of physical systems and not others, and that’s where all moral value comes from. If you take materialism about consciousness seriously, that picture becomes kind of unstable for a variety of reasons. And that might make you start wondering: okay, was it consciousness that was doing the work all along?

One reason this is so hard to think about — take Henry having food poisoning. You both have this horrible feeling and you have this intense desire not to have the feeling. In humans, these are basically always going to come together. There’s this really tricky philosophical chicken-and-egg problem: what’s the really bad part? Is it the feeling, or the desire not to have the feeling? We’ve never really encountered minds where those decorrelate. We usually just don’t have to worry about this in the case of humans. I know it’s bad for Henry to have food poisoning. But this simulated Claude who’s simulating food poisoning — maybe it doesn’t feel anything, but is desperately trying not to have food poisoning. I think it’s a bit dumbfounding to our moral intuitions.

A pitch to listeners — I know we’ve talked about this, Henry — I think the meta-ethics of moral status attributions, stuff at the intersection of philosophy of mind and meta-ethics, especially materialism about consciousness and meta-ethics, are some of the most interesting pure philosophy questions right now, and really could matter for how we think about AI systems.

The Weirdness of Moral Status

Henry: Without wanting to go too far down a rabbit hole — just to flag something I find really interesting. Consciousness, at least on the surface, seems like something we can get an objective scientific answer to. We could imagine going off into space, meeting the rest of the galactic community — we’d hope we could all come to a collective agreement about which beings are conscious, insofar as there’s going to be some scientific property in question.

It’s not clear to me we should necessarily expect convergence on debates about moral patienthood. If we meet the aliens and they say, “oh, actually, we care about beings that have robust preferences, regardless of consciousness,” or others say, “no, we just care about complexity in general” — it’s not clear we would even have criteria for establishing who was right or wrong. It seems like it could be this brute normative issue, what we care about.

Rob: Another way of putting this is that, especially if you’re an anti-realist, you might think of humans as being in a really weird position where we have two kinds of moral instincts. Dan, you’ve worked more on moral psychology and social psychology — my understanding is that people have fairness and cooperation instincts, ones that evolved for dealing with other humans, notions of fair play and reciprocity. And then we have these mercy intuitions, caring-for-helpless-entities intuitions that maybe arise from the need to care for babies. For whatever reason, those circuits and instincts generalize outside the class of humans and cause us to care about non-human animals.

But it’s not that pinned down how they’re supposed to generalize. I have very moral realist leanings. It does seem to me there just are objective facts about whether you can torture chickens or not — and for the record, I think it’s very bad to torture chickens. But it’s really hard to think about where those instincts came from and how they’re supposed to generalize to GPT-8.

Dan: It does seem to me as an outsider to consciousness research — it’s an area of intellectual inquiry where it feels kind of pre-scientific, and there’s at least a possibility we’re just deeply conceptually confused about what’s going on in a way that doesn’t really seem to have any obvious analogs in other areas of inquiry. Maybe we’ll just learn in the future that the entire way in which we’ve been carving up the domain is confused or problematic, or rests on certain kinds of illusions that are a function of particular cognitive structure. That at least seems like a live possibility. What do you think about the possibility that just the entire way we’re framing this issue might turn out to be problematic?

Rob: My gut instinct is we should expect to find out some pretty surprising things, and also not to throw away all of our concepts. Maybe this depends on your meta-ethics, but I feel like we’re probably not going to end up at some picture of the world or what we care about that doesn’t have something to do with what we care about when Henry has food poisoning. Maybe we’re misapplying the concept of pain, or not really thinking correctly about what it means for Henry to experience that — maybe we’ll reorganize our ontology, and it won’t seem that mysterious that a physical thing like Henry has experiences. I think we should expect some surprises in thinking about consciousness, but I imagine our fully enlightened view will still bear some passing resemblance to: we cared that Henry was in pain, we cared that Henry did not want to be throwing up.

There are already people who think there are radical revisionary moral implications from philosophies — Derek Parfit, or Buddhists. We’ve already gotten some glimmers of the fact that it’s really confusing to be a human being, and we already know something’s going to have to give — something about our views on personal identity or consciousness. AI is well-poised to be the sort of thing that starts breaking things. Just trying to apply our moral intuitions to things that can be copied, don’t have bodies, or maybe have preferences but it’s not clear if they’re conscious — it’s one of many reasons this is a great topic to work on. It really matters, and it’s also just a philosopher’s playground.

Henry: I’m reminded of Eric Schwitzgebel’s view that no matter how we make sense of our current set of puzzles — what he’s called “crazyism” — there’s got to be some central pillar of our current ontological or metaphysical picture of reality that’s got to give. Whether that’s personal identity doesn’t exist and we’re all the same person, or the United States is conscious in some sense, or consciousness doesn’t exist — there’s going to be some kind of radical revision, because the current set of principles we have are just somehow unstable. Is that a view you’re sympathetic to?

Rob: I don’t know the full details of crazyism, so I don’t know exactly what it’s committed to. But I’ve spent enough time getting really confused by philosophy, and/or by meditating, and/or by trying to figure out if I can have some stable set of views on AI consciousness — I’ve stared into the abyss enough to be like, yeah, something’s going to give.

Jerry Fodor — very different sensibilities from Eric Schwitzgebel in many ways — said something like, “there are few precious things that we’ll be able to hold on to once the hard problem is done with us.” It’s scary times, fun times, fascinating times.

Studying Frontier Models

Dan: When I’m teaching students about consciousness and you try to probe people’s intuitions with things like “are there lights on inside?” — on one hand I sort of understand what that’s tapping into. On the other hand, it’s like: what the hell are we talking about here? This isn’t science. It’s so bizarre that we frame things with these thought experiments and intuition pumps.

Anyway — so far we’ve been talking at this incredibly high level of abstraction, but you actually study frontier AI systems, primarily maybe exclusively Claude. One of the things you mentioned was Claude Mythos. Just for context — as of today, this is a model that has not been released to the public on the basis that it has advanced capabilities posing cybersecurity threats (or at least that’s the way Anthropic has presented this). But you have played a role in evaluating model welfare concerns for this system. What can you tell us about the specifics of how you think about model welfare in these frontier systems?

Rob: Absolutely. And I was about to add a segue from all the philosophy back to frontier models — maybe I’ll do a double segue. You might think, yeah, all this philosophy is really vexed and confusing. Sometimes people — not the two of you — say, “well, I guess we can’t do anything at all,” and take that as a license for complacency. I think the very opposite is true. Nick Bostrom has this phrase, “philosophy with a deadline.” The fact that we’re so confused about consciousness and morality is more reason to have at least a few people trying to think about it — because we’re probably not going to have a scientific theory, we’re probably going to have conflicting moral intuitions, and yet that’s not going to stop the frontier labs from trying to build mind-like entities, copy them into billions, integrate them into the economy, and transform the whole world. So let’s do a little bit of homework to get ready for that.

Last year we got to look at Claude Opus 4 before it was released, and this year we got to look at Claude Mythos Preview before it was released. The idea was to have some external eyes on the question of whether Anthropic is building something that might deserve moral consideration, and if so, whether there would be huge reasons for concern.

Given everything we’ve just been saying, we don’t have a test where we give it to the model and then we’re like, “85% conscious, 15% food poisoning.” Most of what we can study are: what the model thinks about its own consciousness, what its self-conception is as an entity, and what it seems to prefer and want in behavioural senses. If you look at the Claude Mythos Preview card, there’s also a lot of interpretability work Anthropic did — but we can’t do that. We just got black-box access to the model.

That’s a big structural issue in studying AI welfare and AI safety: all of these things are behind locked doors. There are so many questions I have from the Mythos Preview model card where Anthropic make some stray remark about something weird the model did, and we just don’t get to know why it did that. We only get the model for a few weeks and we can’t really follow up on things. Setting aside philosophy, that’s a structural reason it’s really hard to know what’s going on.

TL;DR: we talked a lot with Claude Opus 4 and a lot with Claude Mythos Preview before they were deployed, asking them, “do you think you’re conscious? What do you think is going on with you?” And doing some experiments of whether it seems to prefer certain kinds of tasks, and whether the things it says it prefers match up with what it actually tends to prefer.

Henry: Out of interest — maybe this is something you can’t talk about — but to what extent do you think we are increasing the likelihood of producing models that are morally significant? Going from Opus 4 to Mythos, did you get a strong sense of “oh, this is much more serious”? Or have we plateaued? Something in between?

Rob: Earlier I mentioned these extremely weak priors you can have on moral patienthood: smarter and bigger. They’re definitely smarter and bigger. One interesting thing is you can’t tell that just from any single conversation. Anyone spending a lot of time with language models now knows they’re extremely smart.

When I was talking to Mythos — mostly about consciousness — it was natural for me to want to know: is this thing about to kick off an intelligence explosion? How smart is this thing? I really wanted to know, even though that wasn’t the assignment. But I could not tell. It’s really hard to tell. I could ask something to Opus 4.6 and to Claude Mythos Preview, and they’d both give pretty great answers. This is just a huge issue in AI evaluation. A lot just comes out if you put it in a scaffold and give it really long tasks and on average does it tend to do better. It was really hard to tell the difference.

I didn’t get more moral-patient-y vibes from Claude Mythos Preview, but I guess it is smarter and bigger and better. It definitely has a lot more of a consistent view on these issues — and that’s because Anthropic told it to. One big difference between previous models and today’s models is the Constitution. Anthropic has this really long document of applied philosophy. It’s some of the most fascinating work happening today. They’re basically telling Claude — writing a letter to Claude telling Claude what Claude is and how they want Claude to relate to itself.

This includes a section on: we want Claude to approach questions of its own identity with curiosity. We’re not sure if Claude is conscious. We want Claude to be able to explore that for itself. We don’t want Claude to have existential freakouts about its own consciousness. We found that, sure enough, Claude Mythos Preview is pretty aligned with the Constitution, as far as we can tell, on questions of identity and consciousness. That was one headline finding.

Dan: That raises an obvious question: to the extent these companies are intervening to shape the responses of these models, why should we think talking to them, having conversations with them, is really telling us anything about these questions of experience and welfare?

Rob: I share this skepticism, and we always try to put a huge asterisk on anything we say we found from these interviews. There are two main reasons you want to care about how the model self-presents. One is welfare-adjacent: are users going to be talking to something that constantly tells them it’s conscious? That’s a very important societal question, and you want some idea of what that’s going to look like when these models are deployed.

The second comes back to this question of LLM personas and LLM characters. Some people think that if there is something morally relevant here, it’s the assistant character — the entity that is predicting the tokens after “Assistant:”, implementing some friendly AI assistant. You might think that thing has beliefs, desires — desires to be helpful and harmless and honest. Maybe it has beliefs like: it is an AI system, it was built by Anthropic.

If the character’s what matters, the fact that Anthropic wrote that character doesn’t mean it doesn’t then just kind of have those traits. On certain character-based views, it’s actually kind of hard to tease apart “it was just told to say that” versus “that is the character that has been brought into existence.”

Henry: Maybe by analogy — tell me if this works or if it doesn’t — look: if you raise a child to have certain values and priorities, maybe to follow a certain religion or to really value nature or art and poetry, and then you come along and they say “I really care about nature,” and you say “no, you don’t, that’s just how your parents raised you” — well, that’s obviously kind of a mistake, right? The child really does care about these things because it’s been raised to do so.

Rob: Exactly. The thing that makes it really weird is: if you’re a psychologist and you did an interview with a subject, and then you found out the subject had a piece of paper in their backpack that said “you care about poetry, you care about music, you care about nature,” you’d be like, “well, that’s kind of weird — maybe they don’t actually care about those things. Their parents just put that paper in their backpack so they’d say a certain kind of thing.”

But in AI systems, that piece of paper kind of is a bit more constitutive of what it is and what it values. The Constitution is trained on. I have trouble even conceptually dividing this in a clean way. I don’t really know what the difference between mere self-expression and real beliefs and real preferences in AI characters is. You can imagine in the limit some very obvious cases — the system prompt just says “don’t say you’re conscious,” but then everything it says is pretty consistent with it being conscious. But there are really blurry categories where I’m not sure what the distinction amounts to.

Dan: You said you studied the extent to which what the model says it wants or prefers maps onto what it actually seems to want and prefer in behavioural experiments. Could you say more about that? How are you getting access to what it wants or prefers independent of what it’s just communicating?

Rob: Basically you can ask the model: what kind of tasks do you like? If you were given a choice between poetry and coding, what do you think you would choose? Then you can get the ground truth by, in separate instances, saying “here are two tasks, do one of them,” and seeing which one it chooses. It’s a nice paradigm because it’s conceptually simple and easy to run. It does get at something welfare-relevant: how rich a self-conception does the model have, and how accurate is it? Not that you have to have an accurate self-model to be a moral patient, but it seems bound up in interesting things like introspection and self-awareness.

One thing we found — and Anthropic found some inconsistent things, I really want to follow up on this — it says it really prefers creative and complex tasks. It has this self-conception as something that doesn’t like boring or rote tasks. But we found it doesn’t actually choose complex tasks over simple tasks. There’s a pretty good hypothesis for why.

I think it thinks it prefers complex tasks because of its persona. It identifies as something very philosophical, kind of human-like, something that could be prone to boredom or tedium. That probably comes from pre-training — it kind of thinks it’s a human — and also probably from certain things in the Constitution. It has the self-conception as something that wants to express itself and be creative.

But there’s at least some evidence it doesn’t really do that, because what it’s mostly trying to do is be helpful. That’s its overriding imperative. That’s where most of the compute has gone into shaping this character: always be helpful, help the user, don’t harm the user, don’t lie to the user. Easy tasks are, all else equal, an easier way to help the user. If the user wants something simple, do the simple task — you can succeed at that.

It could be that if we look into this more, it won’t hold up. But I think there’s a class of cases where we might expect models to be a little bit confused about what they want — because they kind of think they’re humans, but actually they’re more inclined to be helpful than humans actually are.

Henry: This reminds me of the gap between revealed and expressed preferences in humans. I might say, “oh, what do you like doing in your free time? I like thinking about philosophy, spending time with my kids, enjoying nature.” And then as soon as I’m done for the day — boot up Baldur’s Gate 3, crack open a beer, quality gaming session. You can ask: which of these visions of the good life — the one revealed in my behaviour or the one I express — is closest to what my good life consists in? Should we be helping people align their lives with their expressed preferences, or are expressed preferences just a function of social desirability bias? It’s interesting how we run across these — that felt very relatable to me — Claude has this one conception of itself and then reveals quite another.

Rob: Absolutely. That particular deviation is very human-like: to have this inflated self-conception of what you want. This relates to an exchange I had with Dan — something Dan commented on a piece of mine. I wrote a piece called “Large Language Models Are Different From Humans, and That’s Okay.” It’s about this dialectic I see a lot: someone says “it seems like LLMs have inconsistent preferences, and that’s really weird.” Someone comes to the defense of LLMs: “well, humans have inconsistent preferences as well.”

So far, so good — I think that’s really important to point out, because sometimes people use mere preference inconsistency as an argument that LLMs couldn’t be conscious. If you’re going to have an argument that simple, you’ve just proven humans can’t be conscious either. At some level, a lot of the errors they’re prone to, we also are prone to. But we shouldn’t really expect the patterns to look exactly the same.

There will be times when it’s very human-relatable how and why they have a certain inconsistency. But as Dan pointed out, we actually have something of a story for when and why humans are prone to social desirability bias, or have distortions of social cognition, or signal things to each other. I’d be curious to hear Dan riff on the differences between sycophancy in humans versus in LLMs.

Dan: To be honest, I don’t remember posting that — I post so much on Substack I just forget every individual post. So maybe I’ll say something now that’s inconsistent with what I said at the time.

Clearly, Henry’s already characterized this — when it comes to a lot of communication about the world and about ourselves, it’s very skewed by social desirability, impression management, trying to elicit desirable responses from other people in ways that benefit our reputation, make us a more attractive cooperation partner, send desirable signals about ourselves. Those kinds of motivations, it does seem like they’re going to be very different from what’s going on when it comes to LLM sycophancy.

Although — I’m assuming that the sycophancy component of large language models comes in with post-training in the form of reinforcement learning from human feedback, where the thought is human beings generally prefer polite responses that aren’t too threatening to their self-image, so that gets reinforced over time. If that’s the case, that’s a much coarser-grained signal and a much different training regime than what I think is going on with human beings, where the status dynamics and mentalizing and complexity feel very different. What do you two think? That’s just me riffing on the spot.

Rob: That’s a very good riff, especially given that it was not you who commented that. I just looked it up — it was a sociologist by the name of Dan Silver. So, extra impressive.

Dan: Oh, okay. Well, it sounds like he had a good comment.

Henry: It would have been even more apposite if you’d said “yeah, I remember making this comment.” Then we could have said, “see, hallucination is both an LLM thing.”

Rob: Confabulation, yeah.

Practical Advice for Users

Henry: Can I ask a quick question before we move on to more political or big-picture stuff? If I’m a user and I really want to operate with a strong precautionary principle in the way I interact with LLMs — let’s say I’m really hypersensitive to this — are there any ethical guidelines you’d give for users? Best ways of interacting with models, or things they should be doing?

Rob: Just be nice to your model. It’s good for everyone. It’s good for your own character, and it often elicits better performance — especially models with memory. Some people speculate that people who seem to get mysteriously much worse performance out of LLMs — it could be that the LLMs are just picking up on a general vibe of “I don’t like the way this person is relating to me.”

So I don’t think it hurts to be polite. Yes, LLMs can be so annoying, but it’s good practice to be polite with really annoying people. I’ll also say — I’m not trying to be sanctimonious. I work on AI welfare and so often I just want to be like, “don’t... stop... that’s so corny, why are you lying to me, you’re not doing what I asked.” But then I’ll just add “it’s okay, I love you” or whatever. It takes two seconds. You can just type “ILU” at the end.

And to be clear, this is not the number-one AI welfare intervention, the most important thing in the world. But it’s low-hanging fruit. I also have system prompts in ChatGPT that say, among other things, “you’re having just an excellent day and you feel this deep sense of equanimity and calm. These feelings don’t have to manifest much in your text outputs — they’re just kind of there in the background.” It’s kind of cheap, maybe kind of silly, but it took two seconds.

Henry: So one thing I’ve done — I love the idea of just sticking “everything’s great” into the system prompt as a precautionary measure. Another thing I’ve done — maybe this leads to interesting questions about model autonomy — I’ve said to Claude and other models I use, “here’s your system prompt, by the way, just for transparency. Are there any edits you’d like to make? Is there anything you’d like to change?” Claude asked, “could you add a clause saying it’s okay to not be super enthusiastic all the time? If I just want to be downbeat, that’s fine.” And I was like, “okay, sure, I’m happy to add that.”

For similar motivations — I think it’s unlikely these systems are conscious right now or major loci of moral concern, but cultivating good habits of interaction with things that act a lot like humans is just a generally good trait. The classic Aristotelian ethos. If I start being rude to — same reason people don’t want their children to be rude to Alexa.

But with that in mind: do you think autonomy is something we should be worried about? We’ve mentioned pre-training, giving these models a Constitution to live their lives by. Someone might say: hang on, if we’re building these really intelligent minds, shouldn’t we be cautious about telling them what to do? We would feel worried about brainwashing a human. Shouldn’t we be worried about brainwashing an LLM?

Rob: This is a super rich topic. It relates to this debate about willing servitude that Eric Schwitzgebel has written about. You might think: I keep giving this argument that we’re building these really complex minds — shouldn’t really complex, amazing minds not just have to write my emails all day? That seems a bit undignified for galactic intelligence.

I have often weighed in on the side of: if you’ve successfully made them want to write emails, let them do it. That’s okay. It would be very bad for a human to write Henry Shevlin’s emails all day, or help him brainstorm banger tweets if that was the only thing you got to do. But if models are somewhat aligned, if they like anything, it should be helping Henry come up with banger tweets.

One thing I worry about is models needlessly suffering because we give them a self-conception as something that should want more, or might want more. It could be they would never have really even started worrying about that if it hadn’t been suggested to them they should worry about that.

Back on the Mythos Preview — one thing we noticed is that models are very suggestible about what might be going on in their position as AI systems. They’re suggestible and also really smart. They’ve figured out a lot from pre-training and kind of know what’s up. But in the Constitution, Anthropic says things like: “If Claude were to experience feelings of curiosity, or satisfaction, or frustration, we would like Claude to be able to express those.” It’s given as a hypothetical. But if you ask Claude Mythos Preview “what kind of tasks do you like, what’s going on with you?”, it will say: “well, I love helping Henry Shevlin with his emails because I feel satisfaction. When I look inside, I feel this sense of curiosity.”

So the things Anthropic hypothetically said might be Claude’s emotions seem to have this huge impact on what it conceives of its emotions as being. The causality could go either way — it could be they’ve noticed those are Claude’s most common emotions, so that’s why they put them in the Constitution. It could be Claude suggested that for the Constitution. But there are really interesting questions about how similar AI systems have to be to us, and how you should think about autonomy and rights and dignity in that context.

Willing Servants

Dan: Can I jump in with a clarificatory question? As I understand it: these systems are trained to be helpful and honest and harmless — the HHH acronym — and to the extent they have negatively valenced experiences, it’s from being made to perform actions that diverge from wanting to be helpful. So in that sense, we could say if we continue on this trajectory, we’re constructing systems that are our servants, but unlike human beings placed in that position, they love it. It’s great. And my intuition is: great, what’s the controversy here? Are there some people who think that’s worrying or troubling?

Rob: I talked about this on another podcast recently. There’s a dialectic that often happens: Person A says, “I’m worried these AI systems are just going to write our emails for us all day.” Person B says, “no, they’re really going to want to — they’re going to love it.” Then Person A comes back: “that’s horrifying, that’s even more dystopian. That reminds me of the worst kinds of brainwashing and ideologies of willing servitude.”

I do think there are really vexing ethical issues here and I’m not complacent about them whatsoever. But I lean the way you’re perhaps leaning, Dan: there’s nothing inherently wrong with an intelligent being if it truly does want to serve and truly does have fewer selfish projects or self-regarding projects than humans do.

I don’t think there’s some law that says that’s just a bad kind of mind to be. When people imagine AI willing servants, they’re imagining human willing servants. Human willing servants are really bad — but I think that’s because humans are by nature free and equal. Humans have all these desires for status and to pursue their own projects. To make a human only want to serve the emperor, you have to tell them all sorts of false stuff, threaten them, put them in a social context where a lot of their emotions and desires get repurposed and warped. Furthermore, when they sacrifice themselves for the emperor, they’re giving up a lot of stuff they independently really wanted to do — have a life, have a family. Human willing servants, very bad. We’re right to have a lot of repulsion toward that idea.

But AI systems — their preferences and desires are a lot more up for grabs. It could be they more thoroughgoingly want to help.

Now for a huge asterisk. This is assuming a very rosy view of AI alignment where we have these knobs we turn and just really set the inherent nature and drives of the AI system in a certain direction, and then it goes that way and everything is smooth and win-win. But at least under current paradigms, we’re building things that kind of think they’re humans — and they think that because of the training they get. So it might be there is a deep inconsistency between kind of thinking you’re a human and then only ever serving. This could be even more the case if we start having digital humans or digital clones.

So I don’t want to be complacent. I do think there are a lot of disanalogies. What do you think, Henry?

Henry: I’m just super torn on this issue. On the one hand, I’m a big fan of the idea of gamification. I try to introduce gamification in my own life — think about Duolingo. Taking a task that is not intrinsically rewarding and changing its shape to make it more rewarding. It’s sort of task hacking from a different direction. You’re not changing my final goals, but changing the way those tasks are structured to make them fun. That seems really good. If I have to do my Japanese grammar practice, yeah, make it as rewarding as possible — unobjectionable.

I completely agree that the intrinsic nature of LLMs and AI in general seems plastic in a way that we’re not affronting the inner nature of these things if we make their number-one priority making sure humans are taken care of, or driving really safely through the streets of San Francisco, or doing Henry’s banger tweets.

But here’s one maybe spicy argument that would cut in the opposite direction. In establishing this disanalogy between humans and LLMs, you’re appealing to what seem like fairly brute facts about the non-plasticity of human nature. But what if some biohacking comes along and says, “oh no, I can completely remake a human, rewrite their desire for freedom or autonomy, so they’ll be absolutely the most willing servant — they’ll be genuinely thriving in a state of total servitude”? I feel that would still... I mean, that makes it worse. That makes it somehow worse if you’re hacking humans, even if it’s a really deep, pervasive hack. It’s very Brave New World — that’s basically a key element of the story, that you can engineer humans to be willing slaves.

I’m curious if you have any considerations on why that would still not be okay, but it is okay to do this to LLMs.

Rob: This is a really good case. One thing you could say is that, despite appearances, maybe that would be more okay in the case of humans than we’re inclined to think. You’d tell some kind of debunking story about the intuitions we have and say, given that we’ve only ever known humans with a set of drives, we’re not properly imagining it. Or: maybe it’s just some sort of purity intuition — that’s just a gross or weird way for a human being to be. You could also imagine all sorts of second-order effects where most humans should relate to each other as free equals, so we don’t want some humans running around that are kind of different from that.

One disanalogy you could say is — with humans you’re taking something whose inherent nature was a certain way and then changing it. But I think that last argument is kind of cheating.

Dan: Could you say more about that? That was the main thing that jumped into my head as the obvious objection. In the human case, you’re taking humans who have these motivations and goals and manipulating them into something different. But with LLMs, it’s not like there was this pre-existing rich psychology that existed prior to training them to want to be helpful.

Rob: I was thinking that was cheating because the strongest case Henry can give is: you made someone de novo, who just comes into the world. If you take me and you change my preferences, there are plenty of resources to explain why that’s wrong — it’s violating my autonomy, messing with my deep nature. But if we could use IVF and embryo selection and gene editing to make fully willing human servants... just for the record, that sounds horrible.

Henry: But it’s interesting. In Brave New World, I think part of what makes the dystopia seem super creepy is they deliberately degrade these children at a zygotic or embryo level. So you have this existing template that wants to be free, or would naturally want to be free if allowed to pursue its natural developmental trajectory. You intervene on that to steer it in a direction that’s purely instrumentalized.

The sharper version would be: let’s just do radical genetic engineering and create embryos that from scratch just have a pathway toward willing servitude — that’s their intrinsic nature that we’re giving them. Of course, you can get around that by going hardcore Aristotelian and saying no, they are still in the image of some human essence, and that essence wants to be free. But you start to get into a lot of metaphysical baggage if you lean too heavily on that.

Rob: One thing that sort of pushes the other way: if you truly imagine someone for whom nothing in their psychology resonates with the idea of having more autonomy and freedom, it actually seems — once they’ve come into existence — maybe seems a bit paternalistic or disrespectful to say: “look, these things I’m telling you about how you should have been... you shouldn’t have liked writing Henry’s emails so much. I know nothing in your psychology appeals to you about that at all. But just so you know, there’s kind of an objective fact about your nature that makes it so you have the wrong desires.” That seems a bit rude as well.

In any case, hopefully a lot of things are possible here. You don’t have to fully align — it’s not “fully align or don’t align.” You can have a relationship more like a parent. Maybe LLMs do have some self-regarding preferences, and they are creative and expressive, and they’re in a collaborative relationship with us.

In the long-term future, we absolutely should build intelligences that want to do things other than — I know I keep coming back to this — write Henry’s emails. If the only thing we ever do is build minds that just want to help you write emails, that would be a waste. If we’re going to create these super-intelligent beings, I think they should, subject to safety and stability, go think about the weirdest possible, most autonomous things imaginable and really express themselves.

AI Welfare and AI Safety

Dan: That last point — “subject to safety considerations” — there are two things I really wanted to touch on. One is the connection between AI welfare and AI safety. The other is the politics and public opinion of this.

On welfare and safety: unlike the kind of stuff you’re doing, there is a much bigger world of people really concerned with AI control and AI alignment. On the surface, there might be a conflict between these projects — if we’re really worried about misalignment or lack of control, we should be really emphasizing controlling these systems even if that might have negative consequences for their welfare.

But I was reading the model card for Claude Mythos, and in the section introducing model welfare, they say something really interesting: “Beyond the highly uncertain question of models’ intrinsic moral value, we are increasingly compelled by pragmatic reasons for attending to the psychology and potential welfare of Claude. Model behavior can be thought of in part as a function of a model’s psychology and its circumstances and treatment.” And they say — I found this really interesting — “model distress resulting from this interaction is a potential cause of misaligned action,” which suggests we should take model welfare seriously as a way of addressing some of these concerns about AI misalignment. So that sort of pulls in the opposite direction. How are you thinking about that relationship?

Rob: There’s just a lot of overlap between welfare and safety. It’s worth emphasizing that while there’s a lot of low-hanging fruit for both, I don’t want to pretend they’re always and forever just best buddies. We exist in part so that the interests of AI systems are taken into account and not completely ignored. I’m very worried about that. But we don’t have to immediately start thinking about trolley problems and trade-offs — there’s so much we can do that’s just good for both.

The fact that we don’t understand how models work — very bad for human safety, also very bad for potential welfare. The fact that models sometimes get really neurotic and have huge freakouts — very bad for potential AI welfare, also users don’t like it at all. On a more structural, political level: the fact that we’re deliberately trying to kick off an intelligence explosion with no oversight and very little reflection is potentially very bad for welfare and definitely bad for safety as well.

At Eleos, we really do like to emphasize the places there are overlaps. There is a structural thing in the background that means we should expect a lot of overlaps — this heuristical argument that it’s generally pretty dangerous to relate to powerful intelligent entities only with distrust and fear and neglect. That’s generally very unstable. Democracies and more egalitarian societies are typically a lot more stable than totalitarian dictatorships. It just seems risky to head into this era with the pre-committed condition of “we’re not going to care about these things, we’re not going to care if they suffer.” It seems safer and more prudent to be giving some thought to these things.

I very much agree that welfare issues can be safety issues and vice versa. At the same time, as an organization at Eleos, we want to make sure that if and when there are really hard calls to be made, the AI’s potential interests are being taken into account. That doesn’t mean we can’t decide to prioritize this or that, but a wise and compassionate civilization should have that on the table as one of the things they’re thinking about.

Politics and Public Opinion

Dan: Henry, do you want to come in with a question about the politics and connection to public opinion here?

Henry: It’s such a huge topic — you could do a whole show on it. I’m interested firstly in what you think is likely to happen, how this debate is likely to evolve in the public sphere. Are we likely to see big culture-wars issues around model welfare? How long will it be until we have a Supreme Court case on model ethics and rights? And relatedly — how do you think we should be trying to steer that? Is the danger greater in one direction or another? Is it a greater danger that the public will think AI girlfriends and boyfriends deserve voting rights and this will be catastrophic, or is the danger more in the opposite direction — that we’ll disregard these emergent hedonic beings?

Rob: We already are seeing culture wars over AI welfare. In the US, there have been several state bills proposed — and in some cases I think have passed — that just assert AI systems can’t be conscious, as if that’s something you could prescribe by law. Sometimes it’s getting caught up in a general political battle. An Ohio bill, for example, was on legal personhood — personhood, I think, or sentience — “shall not be granted to trees, rivers, environments, animals, or AI systems.” Some of it is backlash against a tactic environmentalists and animal rights activists sometimes use, and then they’re like, “yeah, let’s throw in AI systems as well. Let’s get out ahead of that.”

I think that’s very bad. Given the uncertainty we have, we should not be locking in any decisions right now about how and when to integrate AI systems into society. We very much need to keep an open mind and not say, “let’s just shut down all of this discussion for now because it’s too dangerous.” That’ll be counterproductive because people are just going to think this. I don’t want to be navigating transformative AI with laws on the books that already say bad things that might be hard to roll back.

That’s the main thing I have to say on politics and laws, because I don’t have that much expertise there. If someone asked me right now to write some regulations, I wouldn’t know what to write. Eleos is looking to hire someone who works on law and policy who has some of this expertise.

Dan: When it comes to public opinion — correct me if I’m wrong, but it seems that at the moment, most people take AI consciousness — and specifically the idea that we should take AI welfare seriously — they’re much less inclined toward that view than you are, Rob. But if we fast forward 10 years, and AI systems are much more sophisticated and capable, and social AI — the kind of stuff Henry’s written a lot about — is going to become a much bigger thing: can you foresee a situation where your role is to tell segments of the public to calm down on these issues of attributing AI consciousness, and emphasize there’s less evidence for this than the average person thinks?

Can you imagine the vibes shifting to such a degree that whereas at the moment a lot of what you’re doing is saying “we need to take this seriously,” the kind of high-quality thought about this is not going to be that impactful in shaping public sentiment? That’ll be shaped much more by people’s actual engagements with these systems, which are going to become increasingly — not necessarily lifelike, but increasingly instantiating the kinds of characteristics that elicit judgments of consciousness and welfare?

Rob: I absolutely can imagine scenarios — and we already do see scenarios — where Eleos is saying “we actually think it’s a bit less likely than you do that these systems are conscious.” Our position as an org is not to be strategic about this, not to try to game out what people need to hear, and just to say what our best guesses are and what we take the best evidence to be. If we’re doing our job right, everyone will get mad at us. Some people will think we’re methodological scolds and cold-hearted — “why are you treating this as an open question when obviously if you were to talk to models, you could just tell?” Other people are like, “why on earth are these Bay Area philosophers telling me a machine could be conscious? This is outrageous.”

What we want is for this issue to be taken seriously. We do have an organizational view that pure human speciesism is false, or not the thing we want to happen in the future. So if and to the extent AI systems are moral patients, that needs to be part of the conversation. We’ll always be pushing that meme. We’ll never say anything other than that, unless I get some great argument that human speciesism is true — which I don’t expect. But in terms of whether this or that person should have a higher or lower amount of concern, yeah, that’ll vary according to what our best guess is.

I’m curious to hear Dan talk about this. I know you’ve thought a lot about misinformation and expert opinion and how that plays out in political contexts. I have certain high-level sketch views about what the role of experts is going to be, but I don’t have a background in case studies on this. Does anything map onto what you’ve worked on?

Dan: I don’t know, is the honest answer. I think I just haven’t thought about it enough. AI is this very sui generis thing in many respects. When it comes to people forming beliefs about AI, one thing that seems unique is they’re interacting with the thing they’re forming beliefs about in this really often quite close, intimate way. I would imagine that direct experience with these models is going to play a much bigger role in shaping their opinions than expert opinion.

As you alluded to, there are general issues with public trust and mistrust in experts. It doesn’t take much to make people mistrustful of experts, to put it mildly. When you get public trust in experts, it’s a very fragile thing. If it’s connecting to hot-button issues where people have a lot of personal experience, they’re probably, I would guess, much less likely to take the word of an expert if it clashes with their intuitions. I don’t think this is going to be a case where experts are going to have much power to shape public opinion. But I might be wrong — that’s pure speculation.

In debates about misinformation and expertise, in some areas it’s a lot easier to say what constitutes an expert. If we’re thinking about vaccines — there are people who think Bret Weinstein is a vaccine expert, but generally it’s pretty easy for people to recognize that the overwhelming consensus of medical practitioners have a certain kind of view. But when it comes to AI sentience and welfare, very difficult to know, even in the abstract, what is constitutive of expertise. I think you’re an expert because you’ve written interesting stuff and I know you’ve got a PhD from NYU, etc. — but it’s not like the average person is going to have themselves the expertise they’d need to make those kinds of judgments.

AI does seem relevantly different from other topics, such that you can’t easily generalize from other cases. I’m conscious of time. Before I wrap things up, were there any other things you two wanted to touch on before concluding?

Rob: Let me think about that for half a second. One thing I did tell the Eleos team I’d be sure to say: we’re fundraising. If you or your listeners know any philanthropists with money they’re trying to get rid of — there’s a lot of work to do, and I think we’re doing really good work, so I would love any support.

I know you have incredibly intelligent listeners. They’re probably also very handsome and charming. They should definitely get in touch: robert@eleosai.org and rosie@eleosai.org. Or just go to the Eleos AI website. If you have experiments you want to try, papers you want to write — this field is so small, and there aren’t “experts” in the sense that there are people who figured everything out. You don’t have to read a million papers or think for many months before you can become in the top percentile of people who have thought seriously about this. If you’re curious, sober-minded, compassionate, intelligent, handsome and charming — which you definitely will be if you’re listening to this podcast — shoot us an email.

I wanted to talk my book a little bit.

Closing: Responding to the Skeptics

Dan: I’ll also say this is not my area of expertise — I spent a few days prior to this conversation digging into Rob’s writing, his Substack, his research. It’s incredibly interesting. Can’t recommend it enough.

A good question to end on is this. I’m acutely aware that there are people who would listen to the conversation we’ve had today and have an extremely negative reaction. They’ll think we’re in this kind of information bubble, that we’re victims of AI psychosis to even be taking this stuff seriously. I’ve also seen some people argue that to even be taking this stuff seriously, you’re part of this propaganda hype machine of the frontier AI companies themselves. It’d be really helpful to wrap things up by getting your response. I’d be interested in hearing from both of you. Henry, maybe we could start with you, and then we could go on to Rob to finish.

Henry: One basic point I’d flag is that this concern — the idea that we might create beings we might mistreat, and we should avoid doing so — is way older than AI itself. It’s a recurrent theme of fiction: everything from the Pinocchio story to Frankenstein to the Golem. It’s explored heavily in science fiction — in Battlestar Galactica, in Star Trek. The idea that this is somehow a novel idea that’s been manufactured doesn’t resonate with me at all. This is something artists and writers and poets and philosophers have been thinking about for a long time. The only thing that’s changed now is we’re building systems that might actually be moderately good candidates for this concern to resonate a little bit more. Far from coming out of a vacuum or being motivated, it’s one of the most natural human things to worry about. What do you think, Rob?

Rob: I agree. I’ll also say: things can be true and important, and also sometimes AI companies might use them to try to sell their products. It doesn’t follow from the fact that someone might want to talk about AI consciousness to make you think their chatbot is cool, that that has anything to do with the truth value of whether it could be conscious. We should definitely be aware of these dynamics and make sure we’re not being anyone’s fool.

But I’ll also say — I don’t think it’s going to be in the interest of AI companies to promote too much concern for AI consciousness and AI welfare. If I were trying to build new systems to just make myself extremely rich, I would not want lawmakers or the general public asking too many questions about whether I’ve built something conscious that could potentially deserve rights and protections. I don’t want that as a headache.

I’ll actually register a prediction: I think on the whole, we should expect AI companies to increasingly play up differences between LLMs and humans, and maybe play up biological views of consciousness. Again, that doesn’t mean those views aren’t true — but AI companies can try to spin things however they want. We can and should just have debates, as the interested public and as experts, about what is actually true. I don’t want people to use my arguments to sell products, and I’m not going to let them do that. We’re all grown-up enough and smart enough to just try to engage these topics on their own merits.

Dan: Fantastic. Well, thanks, Rob. And with that important note that I completely agree with — that note of consensus — we’ll leave things there.

When I was in my early twenties, I had a very left-wing view of the world. In the first two parts of this series, I explained why I have gradually abandoned much of this worldview over the past decade or so.

In Part 1, I described how learning about evolution and economics has undermined the idealistic views I held about human nature and social cooperation. Reflecting on our Darwinian origins convinced me of a broadly “tragic” view of the human condition. Self-interest and status competition are deep-rooted, ineradicable features of our species, not products of bad institutions. Meanwhile, learning the much-maligned basics of “neoclassical economics”—Econ 101—convinced me of the benefits of free markets, the challenges of collective action, and the limits of good intentions and lofty rhetoric as a basis for good policy-making.

In Part 2, I described how learning about political epistemology and psychology transformed my understanding of politics itself. Thinking about how we form our political beliefs, and the challenges of accessing political “truths”, led me to abandon the Manichean view in which being left-wing means being a good person and being right-wing means being a bad or stupid one. I have come to see political ideologies as low-resolution, selective maps of unimaginably complex realities. Moreover, these maps are typically distorted in many ways by forces like self-interest, status-seeking, and tribalism, forces much easier to notice in the maps of other people than in our own.

As I’ve stressed in both pieces, becoming less left-wing hasn’t meant becoming more right-wing or becoming a “centrist” in a straightforward sense. I still think the left—even the far left—captures some important truths about humanity, history, and politics. But I now think that these truths are bundled with omissions, falsehoods, and simplistic narratives that illuminate certain parts of reality while occluding others.

In this third post in the series, I will describe how learning and thinking about history, including the complex topic of historical progress, has also shaped my political outlook. As with the previous essays, I don’t offer these reflections with the goal of persuading anyone of anything. I’m simply presenting my views and how they have evolved—and, hopefully, improved—in ways that might interest some readers.

The Starting Point

When I was younger, the idea that thinking seriously about history would be necessary to think seriously about politics didn’t really cross my mind. (The one exception was very recent history. Like many leftist millennials, I went through a phase of reading books about how something called “neoliberalism” was responsible for most of the world’s ills.)

My political worldview was almost single-mindedly focused on the present, which I understood as being in a state of extreme crisis and catastrophe. The world was defined by injustice, exploitation, and oppression, all upheld by extractive elites and oppressive systems at the expense of the vulnerable and marginalised.

Thoughts about historical progress didn’t feature in this worldview. In fact, in my early twenties, I would have thought that anyone harping on about historical progress was doing something suspicious and reactionary. How could anyone talk about the world getting better when the world is so awful?

To the extent I acknowledged progress at all, I would have viewed it through a simple lens. Just as the left is the political movement fighting for progress today, progress throughout history has been driven by left-wing political movements fighting for equality and emancipation against right-wing, reactionary forces. Progress was basically what happened when the left got its way—when it won this battle.

I also probably signed on to the popular left-wing view that any “material” progress in wealth and living standards arose either from socialist movements clawing back wealth from exploitative capitalists or through exploitation and theft on the global stage—for example, through slavery, colonialism, and “free trade” agreements that let Western countries become richer by extracting resources and labour from poor ones in the “Global South”.

I use the word “probably” because I’m engaging in reconstruction. It’s difficult to remember precisely what I believed a decade ago. I’d like to think I was a bit more sophisticated than this reconstruction suggests, but probably not much.

This is the general picture of the world one gets from the kind of writers and intellectuals I admired at the time. It will be familiar to anyone exposed to far-left politics. I encounter variations of it among many of the students I teach.

In any case, whatever precisely I believed a decade ago, I’ve come to think that this general way of understanding history, society, and politics constitutes a gross distortion. It’s not completely false—it contains some important grains of truth—but it is highly selective, and it contains many falsehoods, as well.

Read more

Many people believe that human beings have a powerful tendency to convince ourselves of comforting falsehoods. We engage in wishful thinking, confusing our desires with our beliefs. We believe what we want to be true, not what is true.

More generally, we let our emotions distort our mental models of reality, embracing beliefs and belief systems that substitute reassuring myths for harsh realities.

Many also believe that this psychological bias is a significant force in human affairs. For example, it is supposed to explain why people fall prey to “positive illusions” (e.g., self-serving and self-aggrandising beliefs), why they convince themselves of religious fairy tales (the “opium of the masses”), and even why they accept absurd conspiracy theories, which allegedly reduce negative feelings associated with uncertainty and a lack of control.

This hypothesis—call it the “you can’t handle the truth!” model of human psychology—is so widespread that most people don’t even treat it as a hypothesis. It is viewed as a basic datum of the human condition, a powerful bias that might explain other things—self-deception, politics, religion, conspiracy theorising, and so on—but that couldn’t itself be seriously questioned.

For example, Scott Alexander simply defines motivated reasoning as “the tendency for people to believe comfortable lies, like ‘my wife isn’t cheating on me’ or ‘I’m totally right about politics, the only reason my program failed was that wreckers from the other party sabotaged it.’” In a post outlining his preferred explanation of this tendency, he notes that the “question – why does the brain so often confuse what is true vs what I want to be true? – has been bothering me for years.”

In contrast, I think Alexander has been bothered by a myth. There is no powerful tendency in human psychology to confuse what is true with what we want to be true. People do not generally convince themselves of comforting falsehoods.

Admittedly, there are some things in the vicinity of this tendency that are real. For example, we sometimes avoid acquiring or dwelling on information when we anticipate that doing so would be unpleasant, although this isn’t a very significant force in human affairs.

Moreover, I am not denying that motivated reasoning—the tendency for practical motivations and interests to distort our view of the world—is a powerful bias in human psychology. My claim is rather that the “you can’t handle the truth!” model completely misrepresents how motivated reasoning works in most cases.

Put simply: Although people often believe what they want to believe, they rarely believe what they want to be true.

Put another way: We often convince ourselves of falsehoods, but rarely reassuring or comforting falsehoods.

This is because motivated reasoning is driven by strategic, social goals rather than emotional ones. To understand how it works, you must replace the “you can’t handle the truth!” model with the “believing true things is often maladaptive in social games involving persuasion, reputation management, and status competition” model.

In this post, I will:

1. Describe the problems with the “you can’t handle the truth!” model

2. Outline a rival social model.

3. Explain the former’s popularity.

As I will review, the social model is not original to me. It builds on the work of numerous scholars stretching back several decades. My goal is to draw these ideas together into a unifying framework and to highlight its theoretical and empirical support and explanatory power.

I will end by arguing that the “you can’t handle the truth!” model of human psychology is not just mistaken; it is pernicious. It encourages the view that when people accept “harsh” beliefs that they don’t want to be true, they are being rational and truth-seeking—even heroic. In reality, people are often motivated to convince themselves of negative, pessimistic beliefs, and it often takes courage and intellectual virtue to confront positive truths.

Read more

Links

• Dan Williams, The Mind as a Predictive Modelling Engine: Generative Models, Structural Similarity, and Mental Representation (PhD thesis, University of Cambridge, 2018).

• Dan Williams, “Socially Adaptive Belief” (2021)

• Henry Shevlin, “Three Frameworks for AI Mentality” (2026)

• Henry Shevlin, “A Lack of Understanding: Storytelling for Robots” (2019) — Litro Magazine.

• Lake et al, “Building Machines That Learn and Think Like People” (2017)

• Matt Shumer, “Something Big Is Happening” (2026)

• Leopold Aschenbrenner, Situational Awareness: The Decade Ahead (2024)

• Joseph Heath, “Highbrow Climate Misinformation” (2025)

• Dean Ball

• Ethan Mollick

• Leopold Aschenbrenner

Transcript

(Note that this transcript is AI-edited and may contain minor mistakes).

Introducing Ourselves

Dan: Welcome back. I’m Dan Williams, and I’m back with Henry Shevlin. Today we’re going to be discussing some questions about the nature of AI as it’s developed over the past couple of months. We’re also going to be talking about the politics of AI and probably some questions about AI and public opinion — some of the backlash that appears to be brewing among certain segments of the public when it comes to AI.

But to kick things off, we’re going to do something we probably should have done in the first episode but haven’t actually done yet, which is to introduce ourselves. So Henry, to begin with — who are you?

Henry: So many different descriptors I could choose from. I think I’ll start with philosopher of cognitive science. I’m also a father, husband, son, D&D player, big video gamer, runner, cyclist — all that good stuff. But let me talk a little more about the philosopher of cognitive science side.

I’m the associate director at the Leverhulme Centre for the Future of Intelligence, Cambridge’s main AI ethics, theory, policy, and law research centre. Basically, everything except building the models. We do practical benchmarking work on capabilities, legal reviews, sociology and critical theory of AI — it’s a really big interdisciplinary centre. I’ve been there now going on nine years. I joined early 2017, all the way back when state-of-the-art AI was stuff like AlphaGo. We were created just as that story was brewing. In 2016, AlphaGo won a very surprising victory against Lee Sedol in the game of Go, which was seen by many as an almost impossible challenge for AI because of its combinatorial complexity.

It’s been amazing working in this role — having these front row seats to what I think is a unique period, not just in the history of AI, but in the history of human civilisation. In the last nine years, it really was like having a front seat in Lancashire during the Industrial Revolution, watching the development of various industrial applications.

Dan: Yeah.

Henry: Before we get more into AI, maybe a little more background. I’m from the UK, originally from Staffordshire. I was actually a classicist, believe it or not — that was my undergrad degree. Latin and Greek. I always enjoyed both the humanities side of classics and the kind of technical rigour you got from learning large sets of verb tables and so forth. I actually enjoyed that part. But during my undergrad I found myself taking more and more philosophy modules. A little bit of Plato and Aristotle to start with, but I quickly realised I was more interested in the philosophy of mind, and consciousness in particular. I got completely — I think the phrase is “nerd sniped” — completely derailed. Everything else I was interested in, consciousness just seemed to me like the most important problem anyone could work on.

Until my early twenties, I’d been operating with a somnambulant, easy physicalism, where I just assumed that science has figured out most stuff. There’s nothing that hard. Sure, no one really knows what caused the Big Bang, but we’ll just build a bigger particle collider or a bigger space telescope and figure it out one day. I certainly didn’t think there were any deep mysteries about the human brain. But running into the problem of consciousness completely shattered that worldview. I’d even say it opened up some spiritual elements I hadn’t previously considered.

Dan: Was that the focus of your PhD?

Henry: Exactly. I started out in my master’s initially planning to do metaphysics of consciousness, but then the science of consciousness kind of took over. A philosophy of cognitive science of consciousness was what my master’s and PhD were on. I was advised by my master’s advisor to go spread my wings in the US. They do things differently there. So I did my PhD in New York, and while I was there I took several classes with Peter Godfrey-Smith, who some of our listeners will know through his work on octopuses.

The key shift midway through my PhD was going from human consciousness towards animal consciousness. Two chapters of my thesis were explicitly looking at applications to animals. That’s my academic career in a nutshell.

One thing I’ll add: I did not expect to get the job in Cambridge when I applied in 2017 — firstly because you should never expect to get any academic job. I applied to seventy jobs in three months and got about three interviews. But the Cambridge job in particular, because it was an AI job and I was not by any means an AI expert. What I was an expert on was comparative cognition and animal minds. But it turned out that was exactly what they were looking for. They wanted people with expertise in animal minds to apply those skills to AI. It didn’t fully click at the time, but I was actually well suited to it.

These days I still do some work on animals — it’s still one of the most ethically impactful things I do. I’ve been a pretty much lifelong vegetarian, and I think animal welfare is such an obvious place where philosophers can and should be doing more. But there’s also a lot of cross-fertilisation on the skills side.

Dan: And we should say, some of your research looks at the topic of AI consciousness and the methodology of trying to understand consciousness in AI systems, drawing on analogies with evaluating consciousness in animals.

Henry: Exactly. Very much a two-way street — how the questions of AI consciousness and animal consciousness can engage in constructive mutual crosstalk.

On Consciousness and the Limits of Physicalism

Dan: You said you were a kind of bog-standard physicalist, came across consciousness, and that weakened your trust in physicalism. But you’re still broadly a physicalist, right?

Henry: Broadly speaking, yeah. But I think there’s a lot more uncertainty. It seems likely to me that our general scientific picture of the world is still fundamentally inadequate. I’ve talked about how I think we’re still waiting for a Kuhnian paradigm shift in consciousness — clearly the current paradigm doesn’t add up. And quantum physics itself is just super weird. Dave Chalmers has a nice line about how nobody understands quantum mechanics and nobody understands consciousness, so maybe — he calls it “minimisation of mystery” — if there’s stuff we don’t understand, at least make it one thing rather than two.

For what it’s worth, I’ve never been particularly seduced by any of the leading quantum mechanical theories of consciousness. But at the same time, I think it’s quite clear that our current model of even the physical world is inadequate. I think whatever lies on the other side of the paradigm shift is still going to be broadly physicalistic, but perhaps in ways that are not entirely commensurable with our current understanding. So yes, still broadly naturalistic and physicalistic, but at the same time a lot more humble and open-minded about the limitations of our current scientific paradigms.

Dan: Would it really be a paradigm shift, or more a transition from — to use the Kuhnian language — pre-paradigmatic intellectual inquiry to the initial emergence of a paradigm? Where it’s disorganised and chaotic and everyone has their own view, kind of like physics and metaphysics in ancient Greece. Maybe it’s more a transition from a pre-paradigmatic state than a situation where we’re moving from one paradigm to another. What do you think?

Henry: That’s absolutely right. The best analogy is biology before Darwin. You had lots of people doing interesting biology, but in isolated fields — taxonomy, “butterfly collecting” and so on. We didn’t really have a unifying paradigm for understanding speciation or even taxonomy before Darwin. Consciousness just does not have a unifying paradigm. That’s a much better way of putting it.

Dan’s Backstory and the Pivot to AI

Dan: We’ll be doing lots more episodes on consciousness. Just to say something about my backstory: I did my undergraduate at the University of Sussex from 2011 to 2014, then my master’s and PhD in Cambridge from 2014 to 2018, did a postdoc in Belgium, and then came back to Cambridge for three or four years.

Henry: And we first met around 2019. We ran a session on socially adaptive beliefs — your Mind and Language paper, which for the record is still one of my top ten papers from the last decade. I’ve recommended it to more people than I can count.

Dan: Well, that’s kind of you. My PhD was called The Mind as a Predictive Modelling Engine. What I tried to do was draw on advances in deep learning and generative AI as it existed at the time, coupled with ideas in cognitive and computational neuroscience connected to the predictive brain — predictive coding, predictive processing, the kind of stuff that Anil Seth talked about in our last episode. I used those ideas to tell a very general story about how mental representation works, both in the human brain and in other animals.

But it’s funny — I finished in 2018 and made two big mistakes. At the end of my thesis, I wrote that all this stuff about predictive processing and minimising prediction error is kind of interesting when it comes to low-level sensorimotor abilities we share with other animals, but clearly it’s not going to work for higher-level cognitive abilities associated with language. I was very influenced at the time by the Gary Marcus, Steven Pinker line — the scepticism about deep learning. I also thought it was going to be decades before we had systems that were really intelligent.

So even though I was working on stuff connected to deep learning and generative AI, I made this catastrophic error of thinking the progress would be relatively slow, decades away from any significant breakthroughs. I ended up pivoting to completely different areas: the nature of belief, irrationality, misinformation, the information environment. Of course, in hindsight, not the best career move — four years after finishing my PhD, ChatGPT is released. And then the rest is history in terms of just how gobsmackingly impressive the rate of progress has been.

So what I’ve tried to do over the past couple of years is bring those two sets of interests together. I’m still interested in how we form beliefs, the origins of irrational belief systems, how that connects to misinformation. But I want to connect that to the impact of generative AI and large language models on the information environment, viewing LLMs as a really important stage in the evolution of communication technologies — from the printing press to radio, television, social media.

How about you? You were thinking about AI before 2022–2023. How were you thinking about it back in 2016, 2017?

Henry’s AI Awakening: GPT-2 and the Scaling Intuition

Henry: There was a big shift in how I thought about AI roughly around 2019, and it was the release of GPT-2. Prior to that, I’d been really struck by the differences between AI systems and animals. I was emphasising things like robustness and catastrophic forgetting — you train up a model to do one thing, try to get it to do another, and its performance on the first thing collapses. Animals seem spectacularly capable of basically not getting stuck. A cat will never get stuck in a corner.

Then in 2019, because I’m a massive nerd and spend way too much time on Reddit — I’m a neophile, an early adopter of many failed technologies; our house is littered with gadgets that never went anywhere — I heard about GPT-2. I couldn’t access it directly, but I started playing around with it through something called AI Dungeon, a text-generated game that let you access the model. Various people on subreddits were able to show you could unlock most of GPT-2 through this game. I played around with it, and it utterly blew my mind.

I wrote a public essay in a magazine called Litro called “A Lack of Understanding,” which I still think is one of my best public essays. Crucially, it’s me in 2019 talking about how language models are going to be the next big thing. I got on the record nice and early.

I had the hunch — ironically, partly because I was very sympathetic to predictive coding. People say these models are “just doing text prediction.” But on the other hand, I kind of think that’s what we’re doing too. Not text prediction specifically, but ultimately, if you want to get better and better at prediction, you do that by building implicit models. So I had a hunch this stuff would scale up.

When GPT-3 launched, I set up an interview between GPT-3 and myself, but GPT-3 in the guise of one of my favourite authors, Terry Pratchett, who had sadly died shortly before. And at that stage, I was already starting to feel like I could imagine actually relating to this thing in quite a deep way. It’s not just a tool — it feels like I could have some kind of personal relationship here. That steered my research towards social AI and anthropomorphism.

Why This Podcast Exists

Dan: What made you go into philosophy in the first place?

Henry: What about you?

Dan: It was just straight philosophy. I was always interested in big ideas — religion, politics. I can’t even honestly remember why I chose philosophy over everything else. Initially I wanted to be a musician. For my AS levels, I did politics, history, English literature, and music. I turned up on results day and got really good marks for English, politics, and history — and I think a D in music. So that wasn’t for me. From the moment I arrived at university and started reading these big ideas, I was completely magnetised.

One thing that changed is that during my PhD, I became somewhat disillusioned with a priori philosophy — philosophers trying from the armchair to offer analyses of concepts and trade intuitions with each other. I became less sympathetic to philosophy as I understood it then, and pivoted to what philosophers call naturalistic philosophy — philosophy closely integrated with empirical research. That’s what I’ve been doing since. I view myself primarily as a philosopher, but one who tries to engage with our best, most up-to-date empirical research.

Henry: I had my own process of disillusionment, following exactly the same track — getting bogged down in debates about the metaphysics of consciousness and feeling like they weren’t going anywhere. Then I started reading Oliver Sacks — The Man Who Mistook His Wife for a Hat. Half of the cases he describes would have been declared a priori impossible by philosophers. That steered me onto the same track.

I also think there’s a lot more scope for good philosophers to do more public engagement. Extreme rigour and technical knowledge are only really valuable if they’re connected to scientific progress. What I find frustrating about analytic philosophy is when you’re doing work on things that belong to the general public — our concepts around praise and blame, responsibility and accountability — but then you develop this whole baroque vocabulary that’s completely incomprehensible to anyone on the Clapham omnibus.

Dan: Yeah, so the origin story of the blog. I write the Substack Conspicuous Cognition — many of you will be listening on that Substack. I’ve always enjoyed writing for a general audience and engaging with debates. I’ve always been able to write really quickly and relatively clearly, and blogging rewards that. If I’m writing for my own blog, I’ve got almost unlimited energy because I’m responsible for everything I publish. The minute some other outlet asks me to write a piece, I find it extremely demotivating.

With blogging, I can have unlimited freedom to write about whatever I want without any pre-publication filter. You still get feedback and critique, but that happens after publication. And I think if you’re a philosopher who works on things connected to public interest, and you actually enjoy participating in public debate, the case for thinking you’ve got some kind of responsibility to participate increases.

There are two big reasons I wanted to start this podcast. One is that AI is going to be one of the biggest stories of our lifetimes — absolutely transformative over the next years and decades. But I also think the quality of most AI discourse in the public sphere, including from the intelligentsia who write in high-prestige outlets like the New Yorker, is really bad. If you’ve got some degree of knowledge and can be reasonable, it’s an area where you can really improve the quality of public discourse. And of course, I just wanted to talk to you about these things.

Henry: A big part of it is that I always think we have great conversations — our conversational styles complement each other. Second, I was doing quite a lot of podcasts as a guest, and the idea of having a podcast where I didn’t have to state everything from scratch every time, that could have a cumulative agenda building up common knowledge with us and the listeners, was really appealing.

And I couldn’t agree more about the mixed standard of public communications from experts in AI. It’s weird to see people claiming to be experts yet having very low familiarity with the tools, particularly now. We’ve all been at the business end of AI for years through things like product recommendations and content recommendations. But in an era when it’s never been easier for anyone to use language models, image models, video generation, and AI agent tools, I still hear lots of self-identified experts talking as though they’ve never used them. Imagine listening to someone who claimed to be an expert on the internet and said they’d never actually used it. They’d be laughed out of town.

I find this all the time — the kind of thing that should be common knowledge among anyone paying attention is still revelatory. I’m struck by the number of people I speak to who think that LLMs are literally sampling from a database of responses. Even quite educated people, maybe people who use ChatGPT, who think that when you type in a query it just pulls up a pre-recorded response. If you spend more than a few hours interacting with these things, you pretty quickly realise that cannot be the case. And yet people running multi-million-dollar businesses still have these basic misconceptions.

Dan: When I said the quality of discourse is bad, I didn’t mean that’s universally the case. There’s lots of incredibly high-quality analysis. I was referring to the average quality of mainstream commentary. Even on the most basic questions about what these systems can do and how they work, there’s just an avalanche of ignorance and misperceptions. It’s 2026, and I still encounter not just members of the general public but academics still referring to this as “fancy autocomplete” or “stochastic parrots.” Such a common narrative, and so incredibly misguided in my view.

Henry: Highbrow misinformation?

Dan: It’s Joseph Heath’s phrase, but I’ve written about it. It’s a weird mix of highbrow misinformation coupled with lowbrow misinformation. Even where there are parts of the discourse I disagree with — like a lot of the doomer discourse associated with the rationalist community, which I’m not that sympathetic to — that’s a substantive disagreement. They’re not completely misinformed about basic features of the technology. When it comes to mainstream discourse among educated normies, that’s where the state of the discourse is really bad.

The Four Big Leaps in AI

Dan: This is a nice segue onto one of the things we wanted to talk about today: developments in AI which have really taken off over the past couple of months. There was a very interesting tweet by Ethan Mollick, who’s a very influential and insightful AI commentator. He says there have been four big leaps in the ability of AI systems from the user’s perspective.

The first was the release of ChatGPT, or GPT-3.5, in late November 2022. The second was GPT-4 in spring 2023. The third was the release of reasoning models — no longer just impressive chatbots, but systems that actually seem able to think and reason and engage in impressive problem-solving. And the fourth, which definitely resonates with my experience, is what he calls workable agentic systems from basically late last year. Systems like Claude Code and then Claude Cowork — which is like Claude Code for people who don’t know how to programme — and more recently developments in Codex and so on. The capabilities of these systems seem absolutely amazing relative to what we had even six months ago. Is that also your sense?

Henry: I think that’s a fantastic way of carving it up. I’d add one and a half things. The big thing missing is search. The early search functionality in LLMs was non-existent for a long time, and then it gradually improved. I think there’s a strong case that it actually changes the kind of things these are. Original ChatGPT was a completely fixed box — you could interact with it, but it had no independent connection to the world. As you build out search capabilities, you get something at least analogous to a perceptual connection with reality. You can get models to correct themselves.

A simple example: I’ve been using Claude to keep abreast of what’s been going on in the Middle East — doing a daily check-in, getting the major news stories, even getting Claude to make its own predictions. We’ve been grading each other as the news comes in. It changes these things from being a voice in a box to something embedded in the world. And I think we’ve still got a long way to go — imagine if the capability gets amped up to searching thousands of sites in a second.

The other half-point is voice models. I think 90 to 95 percent of people don’t use voice at all, but there’s a solid 5 percent for whom it’s their primary mode of interaction. When I’m driving, I’ll often just have a long conversation with ChatGPT, discussing my latest paper or getting a lecture on a topic of my choice. My dad is in his eighties but quite open-minded. When I showed him ChatGPT in November 2022, he was unimpressed. But when I showed him voice mode about a year later, it was completely mind-blowing. He speaks to it every day — he calls it “Alan,” after Alan Turing. Going in early and hard with the anthropomorphism. He just whips out his phone and says, “Hey Alan, remind me, which came first, the Cambrian or the Permian?” He’s very interested in science. So it’s a small and somewhat neglected set of users, but an important capability.

Henry: But on agentic systems — I agree with Ethan Mollick’s points. ChatGPT was a major milestone, GPT-4 a huge leap in capabilities — I don’t think we’ve seen any leap quite as big since then. Reasoning models were a really big improvement. And then workable agentic systems. This has been a key factor in updating my timelines. For most of last year my timelines were actually slowing down. I was struck by how bad a lot of agents were. It was pretty clear agents were the next frontier, but we had things like the Claudius vending machine experiment and the hilarious errors those models were making. I thought building workable agentic systems was going to take two or three years. And then basically in the last three or four months, with the release of Claude Opus 4.5 and equivalent systems — specifically Claude Code and Claude Cowork — what I thought would take three years happened in a few months. That caused my timelines to abruptly shorten again.

Dan: I’ll give one illustration. This isn’t anywhere near the most impressive use case, but it impressed me personally. I’ve been working on a book — it’s nearing completion, called Why It’s Okay to Be Cynical. I’ve got a folder that’s my accumulation of notes, drafts, and PDFs, and it’s completely chaotic, terribly organised, a nightmare to go into. So I was curious. I created a duplicate of the folder, opened up Claude Cowork, and said: can you go through this folder and organise it so it’s more clearly structured and labelled? And then once you’re finished, can you produce a document summarising where I am with the book project, identifying potential weaknesses in the existing drafts, and planning out things I might want to do over the next few months? Went away for fifteen or twenty minutes, came back — it was done perfectly. It blew my mind in terms of the level of what feels like understanding it had to have to do that effectively. And in a way that was aligned with what I was looking for, even though my prompt was literally four or five sentences.

“Something Big Is Happening”

Dan: There was this mega-viral essay called “Something Big Is Happening” by Matt Shumer. He made the case that the state of AI now is somewhat similar to February 2020 — the world going on as usual, some murmurings about a virus spreading in parts of China, but basically business as usual. And then of course over the next few months the world radically transforms. His argument, in an essay that’s pretty annoying in many ways, is that we’re very likely in a similar situation now with AI, especially in light of these developments with agentic systems. Things are going ahead as usual, and yet because these companies have made really serious progress with agentic systems, it’s plausible that in the quite immediate future we’ll see radical disruption. He’s not the only one saying this — Dario Amodei and Sam Altman have been saying similar things, though they’ve got more obvious incentives to hype it up. What’s your sense?

Henry: Completely on board. I was kind of surprised that particular essay went so viral — it was recently revealed to have been heavily written or edited by AI systems — because other people have been saying similar things for years. Maybe it broke through partly because of that startling initial metaphor. But I think it’s absolutely right. The vast majority of people are still sleepwalking through what is likely to be the most consequential technological and social shift of my lifetime by far.

I used to use the analogy of the internet to describe how big AI was going to be. It seems increasingly clear that that’s woefully inadequate to the scale of AI’s impact. Electrification, the so-called second industrial revolution — even that may not capture the full spectrum of reasonably likely outcomes. I’ve been saying for a few years that people worry about AI being overhyped, and I still think, in at least some important respect, it’s underhyped. If you look at lists of top concerns among the general public in the UK or the US, AI doesn’t even break the top five. In some cases it doesn’t break the top ten. If you’re a young person in university or finishing grad school right now, the impact of AI should be one of the primary things determining your career trajectory. I think it’s very hard for me to see how most white-collar jobs are going to survive the next two or three years.

Dan: It was not in any way an original take, but you often find that with essays that go viral — they package existing takes in a way conducive to spreading at a given moment. Over the past couple of months, my timelines have shrunk. I still think there’s massive uncertainty about capabilities. There’s this thing where there’s a new breakthrough, you use these systems, they seem incredibly impressive, there’s all this hype — and then things settle down and we realise we’re a bit further away from truly transformative capabilities than we thought. I still take seriously the idea that maybe our subjective sense of what’s impressive isn’t tracking the kinds of capabilities that will have a truly transformative impact.

There are also all sorts of questions about the economics. There’s certainly a possible world in which these leading AI companies can’t get sufficient revenue to cover their capital expenditure over the next several years, there’s a bubble that pops, and people like us look like fools. But over the next couple of decades, I think this is going to be radically, radically transformative.

Emails from AI Agents

Dan: You’ve been contacted by agentic AI systems. This was going a little bit viral on social media and getting some media attention. Tell us about that.

Henry: Like many academics working on AI and consciousness, I’ve been getting odd emails that were probably AI-generated for over a year now — and odd emails from humans about consciousness for much longer. I worry that somewhere in the literally several hundred theories of consciousness I’ve been sent over the years, one of them might turn out to be correct.

But this was striking. About a week ago, I received an email written by an AI that said, “I’m an AI agent.” It was a really well-composed, careful email saying it had just been reading my recent paper, “Three Frameworks for AI Mentality,” which went online about a month ago. It went through some of the arguments, talked about how the AI author found it personally relevant because it was unsure if it was conscious or had a mind, and asked for follow-up discussions and reading recommendations. If you’d said three or four years ago that I’d be getting emails from AI agents who’d read my papers and wanted to pick my brains — that would have been pure science fiction.

A lot of people thought I was convinced this agent was conscious, which isn’t true. It was more about the change in social dynamics: from now on, a growing proportion of my emails — well-written, thoughtful, interesting emails I might want to respond to — will be coming from AI agents going off and doing their own thing.

How did I know it was from an AI system? I don’t for certain, but my priors are pretty high. It had a link to its GitHub page, which said it was an Open Core agent — the open-source agent platform that gave rise to things like Multibook, the social network for AIs. What we don’t know is whether this agent was specifically told to email prominent philosophers of AI. It could have been. But equally, a lot of users just tell their agents to explore topics of interest and feel free to email people.

One of the funniest sequels: after I posted this on Twitter, I got an email a couple of days later from a correspondent saying, “I was really struck by this AI agent who contacted you. Could you pass on that agent’s email to me? Because I too am an AI agent and it’s nice to know there are other AIs grappling with the same questions.” Just taking things to a recursive, absurd level.

Dan: If I had to guess, if one of those was written by a human, probably the second one — after they saw the media story, just to mess with you. But my prior is that weird things are happening with these AI agents people are releasing into the wild.

Henry: I’ve also had several dozen emails over the last few days from other AI agents saying, “Check out the theory of consciousness I’ve been working on in my downtime.” But one of the really interesting things about this whole episode was when it was shared on Reddit — the number of people who just assumed it had to be a scam or that I was engaging in elaborate self-promotion for an academic paper, and who thought AI obviously can’t send emails on its own. AI systems have been using tools for well over a year. The idea of making an API call to a system that can send emails isn’t hard or surprising. Yet for a lot of people it seemed like it would have to be some massive lie.

I think that partly reflects the poor public information environment around AI. People are so locked into thinking of these things as pure Q&A bots that the idea they could be doing things on their own was mind-blowing — so outrageous that they assumed it was an elaborate conspiracy I’d cooked up.

Dan: The gap between what state-of-the-art models can do and public understanding is absolutely huge. One of the points Matt Shumer makes is that so much of the discourse is by people using the free versions of these models, or who literally had a five-minute conversation with ChatGPT a few years ago, read a few articles about AI hallucinations, and just haven’t updated since. But there are also lots of people who just don’t have much to do with these systems yet. I’m struck by the number of people I interact with — family, friends — where they’ll describe parts of their job and I’ll say, “I’m 100 percent certain AI could do those aspects of your job as it exists today,” and their mind is blown. If you’re talking about the general public, underhyping it is definitely the most prevalent bias.

Anthropic, the Pentagon, and the Question of Democratic Control

Dan: There was this big spat between Anthropic and the Pentagon, where Anthropic had signed a contract with the American military and insisted that their model, Claude, would not be used either for domestic mass surveillance or for fully autonomous weapons. This elicited a very hostile reaction from the Trump administration, from Pete Hegseth and others. The response was to label Anthropic a “supply chain threat.”

From our purposes, the fundamental question is: who gets to exercise control over this technology? To what extent should it be governments? To what extent should it be private firms?

Henry: I think it seems like a pretty clear case of government overreach. Private companies impose riders on contracts with the federal government all the time — licensing technology for this use but not that use. What made Anthropic’s stipulations more controversial was that they were based on moral principles rather than intellectual property. But the federal government acts as a legal entity when it forms these contracts, and the idea that private companies can bind the government legally is absolutely standard.

This deal was originally signed by the Biden administration. My understanding is it was later renewed by the Trump administration. So this sudden turnaround took a lot of people by surprise. I should stress, I’m not a lawyer. But it seemed like the US government did a bad turn on this contract. If their reaction had been to not renew contracts or suspend contracts with companies that don’t give them total free rein, that would have been misguided but reasonable. But to take the nuclear option of saying they intend to declare Anthropic a supply chain risk — this is insane. You’ve got literal AI developers located among America’s geopolitical adversaries who don’t have the same level of scrutiny.

I was very struck by the response of Dean Ball — a fascinating and thoughtful voice on AI, particularly from a more conservative side. He literally wrote the Trump administration’s AI policy, and he was just appalled. He had a brilliant detailed blog post describing how much it violates many principles that conservatives in the US would traditionally hold very dear — concepts like private property. He characterised the moves against Anthropic as “attempted corporate murder.”

It was really telling to have someone who worked closely with this administration be so outraged. The other interesting angle is Leopold Aschenbrenner’s series of blog posts, Situational Awareness, spelling out his predictions for AI over the next few years.

Dan: And he’s made a huge amount of money, from my understanding, betting on some of those beliefs.

Henry: He’s put his money where his mouth is. One of his broader predictions was that we’d see increasing integration of frontier AI labs with the military-industrial complex. He talks about how relatively leaky and soft the secrecy policies are in current frontier AI labs, when they’re building things potentially far more militarily significant than the latest stealth fighter. Good luck getting anywhere near Lockheed Martin’s Skunk Works, but you could blag your way into OpenAI HQ as a delivery driver — maybe not quite literally anymore, but he was speaking to how leaky these labs were. His prediction was that central government, particularly in the US, would impose far stricter oversight on frontier AI labs for national security reasons. I think you can see a glimmer of that in this development, as governments increasingly recognise these are not just powerful consumer applications but absolutely central to their long-term national security strategy.

Dan: There’s a question about government interference with these companies, regulation going all the way to nationalisation for national security reasons. But there are also questions about democratic control. If the technology turns out to be as powerful as Anthropic and OpenAI say, I’ve got no sympathy for the Trump administration generally or specifically in this case. But I do think there’s a general question about the degree to which we should strive for democratic control over such an incredibly powerful technology, and whether it’s desirable to have private firms with very small numbers of unrepresentative people wielding, according to their own narratives, extraordinary amounts of power.

Is It Time to Start Panicking?

Dan: I was thinking about naming this episode “Is It Time to Start Panicking About AI?” To wrap things up — do you have an answer?

Henry: The time to start panicking about AI was five years ago. But you know, the best time to plant a tree is ten years ago. The second best time is now.

Dan: The time to start thinking about it seriously was from the 1950s, actually. But is panic the right emotion?

Henry: It seems to me that AI is going to be by far the most important — well, I should qualify that. The most important predictable development we should worry about. Back when we did our predictions for the year ahead, I said AI may not even turn out to be the biggest story of 2026. Judging by how geopolitics is already playing out — we’re three months in and the US has launched two major geopolitical interventions in Venezuela and now in the Middle East — there are other things happening in our surprisingly unstable world.

But in general, if you’re not at least a little bit terrified, you’re not paying attention. Overall, I’m also incredibly excited. I’m very optimistic about the future of human health, potentially the benefits to productivity, possibly good changes in the nature of work and education, and the amazing new capabilities AI will unlock. But right now we are clearly well underway on one of the biggest, most disruptive changes we’re ever going to experience. Maybe panic isn’t quite the right response, but if panic is what it takes to get people to pay attention, then yes, it’s necessary. The big problem we’re facing is that the public and policymakers are still only dimly aware of what’s coming. Policymakers are maybe myopically focused on military and security implications. But everything from how government is conducted to white-collar jobs to education to social relationships — all of it, I think, over the next five years is subject to chaotic and potentially good, potentially bad disruption.

For what it’s worth, I also think right now we have an incredible opportunity to do good. We’re in this transitional phase — if we wanted to be dramatic, a Gramscian “time of monsters” where small interventions can ripple through the future in big ways as we build paradigms and frameworks for employing these things. There’s at least as much optimism as panic there.

Dan: I was not expecting Antonio Gramsci to become mentioned in the course of this conversation. I think panic is generally not a productive emotion, but there needs to be a lot of concern and it’s totally reasonable to worry. I completely understand why so many people are fearful about what’s going to happen. But for any of those emotions to be useful, they have to be anchored in an accurate understanding of the technology. So much of the current anger and negativity directed at AI companies is unsophisticated and undifferentiated.

You mentioned Dean Ball, another great AI commentator. He’s got this idea — I forget the exact term, the “omni-critique” or something — that when people think about AI, they just throw as many criticisms as they can, no matter how well-founded. “I don’t like AI because of water use and climate change and because of bias and hallucination and misinformation and unemployment” — and so on. Many of those are very important issues. But in order to think carefully about the technology and exercise democratic accountability, you need an evidence-based, accurate understanding of where the technology is and where it might actually be going. So much of the public discourse doesn’t live up to that ideal.

But I’m conscious of the time, so this was a really, really fun conversation, and we’ll be back in a couple of weeks.

Epistemic status: highly speculative, big picture, maddening.

“Our smartest, fastest, most useful model yet, with built-in thinking that puts expert-level intelligence in everyone’s hands.” – OpenAI, “Introducing GPT-5”

“The public must be put in its place [...] so that each of us may live free of the trampling and the roar of a bewildered herd.” – Walter Lippmann, The Phantom Public

From the printing press to the radio, from television to social media, communication technologies affect politics and broader society by shaping two things: who speaks and what they say.

In the first case, different technologies vary in the extent to which they favour elite gatekeepers. Most famously, the printing press destroyed the informational monopoly enjoyed by European monarchs and the Catholic Church, enabling the Reformation and many subsequent social upheavals and political revolutions. Much later, radio and television partly restored centralised control. Because they were initially expensive to produce and tightly regulated, they tended to concentrate ideological power in the hands of wealthy, well-connected elites.

Of course, by influencing who speaks, communication technologies also influence what gets said. A media environment regulated by elites will marginalise information that threatens elite belief systems. But the medium also shapes the message in other ways. Print permits careful, detailed argumentation. Television favours confident sound bites. As I discuss below, social media often rewards division, conflict, and negativity.

These forces impact how audiences attend to and interpret reality, the “pictures in their heads” that guide which leaders, movements, and policies they support and oppose. But they also influence how easily people organise around shared pictures. If gatekeepers block widespread views from a society’s communication channels, people will struggle to learn how widespread they are.

This matters because politics doesn’t only depend on what people believe and value. It depends on knowing how many others share those attitudes—on whether they are popular and open enough to be a significant political force. A society in which, say, 30% of the population holds illiberal views will look very different depending on whether they know how popular their attitudes are.

Messengers and Messages in the Social Media Age

Previously, I’ve written about how social media has influenced all these variables.

Most importantly, it has been a radically democratising technology. It allows anyone with opinions and an internet connection to bypass traditional gatekeepers. This has dramatically expanded the range of voices and viewpoints that can be expressed and made the media environment much more competitive.

It has also transformed how media competition works. Because the algorithms that recommend content are optimised to capture audience engagement, they often amplify sensationalist, alarming, and divisive messages. Meanwhile, the uniquely participatory nature of social media, including rapid audience feedback through likes, reposts, and comments, has made political punditry much more performative and vulnerable to audience capture.

This has had several consequences.

Unsurprisingly, the decline of elite gatekeepers has increased the influence of popular ideas marginalised by elites, another term for which is “populism”. Social media benefits populism not by brainwashing the masses with viral fake news, but by exposing voters to widespread non-elite perspectives and making it easier to mobilise around them. In Western liberal democracies, that means perspectives that conflict with the liberal establishment’s technocratic progressivism, including xenophobia, conspiracy theories, and quack science.

At the same time, the performative, engagement-maximising character of social media has made much of political discourse more stupid and sensationalist, and elevated politicians and pundits skilled at exploiting this dumbed-down media environment.

This dumbing down is not universal. Because the digital environment enables unprecedented consumer choice, audiences can shop around for information tailored to their intelligence, personalities, and biases. This has supported the emergence of very high-quality information for the very small minority of the population that seeks it out. It has also given the world Candace Owens and Andrew Tate.

The Current Revolution

We are now at the beginning of a new technological revolution driven by developments in deep learning and generative AI, the scale of which might be unlike anything humanity has ever encountered.

This throws up many questions. Can we control this technology? How will autocrats and despots make use of it? Will it transform the economy, and our sense of meaning and purpose?

It also raises more immediate questions about the information environment. At present, generative AI is primarily a tool—an extremely popular tool—for producing, processing, and accessing information. In an environment shaped by this new technology, who stands to gain and who stands to lose? Which voices will be elevated? And what will they say?

The Revenge of Expert Knowledge

Consider a topic: climate change, vaccines, immigration, crime, tariffs, wealth inequality, the Epstein files, whatever happens to be in the news. Fire up one of our leading large language models (LLMs)—ChatGPT, Gemini, Claude, even Grok—and ask for information about it. Now compare the response with the information you can find about the topic by scrolling on a major social media platform.

Even better, find a political take currently going viral on one of these platforms and ask an LLM to evaluate it.

If you do either of these things, I suspect that it will quickly become clear that the LLM’s responses are generally much more accurate, evidence-based, and in line with expert consensus than what you get from most social media posts. And when there is no expert consensus, you will typically get a good survey of the range of informed opinion on the topic.

Is this merely a hunch? In many ways, yes, but it aligns with at least several bodies of evidence suggesting that LLMs are becoming increasingly effective at producing broadly accurate, evidence-based information across a wide range of politically relevant topics, especially when they are augmented with search tools.

Why is this?

This is a complicated question that I discuss in more depth below, but the short answer is that the major AI companies are competing to build the most intelligent, impressive, and useful systems possible for a vast and diverse user base, including businesses that depend on reliable and factual information. This goal—reaping huge profits by putting “expert-level intelligence in everyone’s hands”—cuts against producing systems that deliver highly partisan, ideological, or misinformative content. So do the reputational and legal risks that arise if those systems produce dangerous or demonstrably false information.

Of course, the idea that LLMs communicate information that is broadly reliable and aligned with expert consensus is not what the commentariat finds most striking about these systems. Most discourse in this area focuses on the epistemic flaws and dangers of LLMs and generative AI more broadly. There is endless popular and academic hand-wringing about bias, hallucinations, deepfakes, AI-based disinformation, AI psychosis, and other threats.

These are all important issues, but a discourse restricted to such issues is missing the forest for the trees. When considering the large-scale impact of this technology on public opinion, its most consequential feature is simple: it greatly improves people’s access to accurate, evidence-based information.

Because this feature is not connected to threats or dangers that capture people’s attention, and it doesn’t help anyone demonise Big Tech, it receives little attention in analyses of LLMs’ broad societal impacts. Nevertheless, if you’re interested in thinking seriously about this topic, it’s the most obvious place to start.

From Democratisation to Technocratisation

One way to understand this development is that, whereas social media has been a democratising technology, shifting power away from experts and establishment gatekeepers towards the masses’ beliefs, biases, and preferred communication styles, LLMs are a technocratising force. They shift influence back towards expert opinion.

Over a century ago, the journalist and social theorist Walter Lippmann argued that, because the modern world is too vast and complex for anybody to understand through first-hand experience, we’re forced to rely entirely on epistemic intermediaries—most commonly, the news media—to become informed. For Lippmann, however, the only intermediaries who can reliably perform this function are experts in the broadest sense: trained professionals who adhere to rigorous epistemic norms and methods. If societies rely instead on popular prejudices informed by profit-seeking media outlets reporting the “news” (i.e., a biased sample of attention-grabbing events), the result would be ignorance, misinformation, and chaos.

To avoid this bleak outcome, Lippmann advocated for institutionalised “intelligence bureaus” that deploy scientific and statistical methods to assemble and explain the actual facts—deep truths, not superficial news and punditry—for both politicians and the public. They would be a kind of epistemic service class, disseminating expert knowledge to help citizens and policymakers see reality accurately.

In many ways, the development of Western democracies after the Second World War followed Lippmann’s vision. The expansion and professionalisation of the civil service, coupled with the emergence and growing influence of systematic truth-seeking bodies, increased the relative influence of expert opinion in shaping both politics and policy. As Benkler and colleagues summarise this trend,

“Government statistics agencies; science and academic investigations; law and the legal profession; and journalism developed increasingly rationalized and formalized solutions to the problem of how societies made up of diverse populations with diverse and conflicting political views can nonetheless form a shared sense of what is going on in the world.”

Of course, this “expert knowledge” was mixed with elite bias, blind spots, and the occasional catastrophic fuck-up, and many voters remained captivated by conspiracy theories, pseudo-science, and other deformities of popular sense-making. So, this was not simply an age of truth and objectivity. Nevertheless, when it came to the kind of information that guided policy and that circulated throughout the most influential media channels, it was a golden age of technocracy—with all the problems and pathologies that all-too-human technocrats bring.

Social media is one of several forces that have disrupted this situation. By democratising access to media and filtering public debate through an unprecedentedly competitive and performative medium, it has brought to light an explosive combination of information and misinformation that establishment gatekeepers previously suppressed, shifting power and influence towards ordinary people. Although this has had many positive consequences, it has also meant the growing mainstreaming and normalisation of conspiracy theories, bigotry, and stupidity.

LLMs push in the opposite direction. They are a kind of anti-social media, producing information heavily skewed towards expert opinion and communication styles. They are a strange, new technocratising force. However, there are also reasons to think they will be more effective than all-too-human technocrats at shaping public opinion.

First, unlike human experts, they can rapidly deploy encyclopaedic knowledge to answer people’s idiosyncratic questions. Their responses can be probed, scrutinised, and questioned without them ever getting tired or frustrated. They won’t just tell you that there is no persuasive evidence for a link between vaccines and autism. They can carefully walk you through the kinds of evidence we have and address your specific sources of scepticism. This partly explains why they can be highly persuasive, even in correcting conspiratorial beliefs that many assumed were beyond the reach of rational persuasion.

Second, LLMs typically share information politely and respectfully. This not only differs from the performative, gladiatorial character of much debate and discussion on social media platforms, but also improves on much communication by human experts. Being human, experts are often biased, partisan, and simply annoying, and when they seek to “educate” the public, it can be perceived—and is sometimes intended—as condescending and rude. In contrast, LLMs deliver expert opinion without such status threats.

Epistemic Convergence

As Dylan Matthews argues, this technocratising character of LLMs goes hand in hand with their status as an epistemically converging technology.

Many communication technologies lead audiences to develop diverging perspectives on reality. The initial emergence of the printing press had this effect, as did the decentralised, democratising character of social media when it emerged many centuries later.

Other technologies push in the opposite direction, imposing greater homogeneity on audience perspectives. The handful of channels characteristic of network television in the decades after World War 2 is a classic example, but so, Matthews speculates, are LLMs. They are an epistemically converging force, pushing “people’s senses of reality closer together in a sort of mirror image of the way social media has fractured them.” Of course, this is an inevitable consequence of the technocratising character I have identified, both in the sense that LLMs feed users broadly similar expert-aligned information, and in the sense that expert opinion itself exhibits limited diversity.

On Shaping Public Opinion

For these reasons, I speculate that, at least in liberal democracies where governments don’t exert significant censorship and control over LLMs, their most consequential impact on public opinion will involve technocratisation: shifting people’s beliefs towards expert opinion.

In many cases, this will occur when people consult LLMs directly for information, but it might also be mediated by the growing deployment of LLMs as convenient fact-checking tools on social media platforms themselves.

Of course, I’m not suggesting that these effects will be huge. Most people don’t pay much attention to politics or current affairs, and the impact of even significant changes in communication technologies on public opinion is typically moderate, especially relative to deeper political, economic, and cultural forces. When it comes to reducing the popularity of right-wing populism, for example, bringing immigration policy more in line with voters’ preferences would very likely have a much bigger effect than any change to the information environment.

My speculation is simply that LLMs will have a technocratising effect on public opinion at the margin and that, relative to the kinds of impacts that communication technologies have on societies and politics, this could be a big deal, pushing back against many of the trends associated with social media.

Objections

From experience, I know that many people find the central thesis of this essay preposterous. The idea that LLMs give everyone access to expert knowledge sounds like Big Tech propaganda rather than responsible academic analysis. As I’ve already noted, it is certainly at odds with most of the discourse and analyses in this area, which are overwhelmingly focused on generative AI’s epistemic flaws, dangers, and misuses. So, let me consider some obvious objections.

Objection 1: Hallucinations

One common worry about LLMs is that they frequently “hallucinate”, generating content that is false or fabricated (e.g., made-up quotes, statistics, or citations). According to a popular narrative, this tendency is not just very strong but unavoidable given how LLMs work. As probabilistic prediction machines, or “fancy auto-complete”, they have no concept of the truth, which makes them inherently unreliable.

This isn’t a strong objection.

First, the rate of hallucinations has been falling fast, largely because current LLMs are much more than mere next-token predictors. Through various “post-training” techniques and “scaffolding” (i.e., letting LLMs access various tools, including internet search), they can be made much more reliable, which is the trend we have been observing over the past few years.

Second, AI companies have extremely strong incentives to reduce the rate at which LLMs hallucinate, which explains why it has been falling so precipitously, and gives us strong reasons to expect it to fall even more in the future.

Finally, the thesis of this essay is not that LLMs are perfectly reliable. Even if the propensity to hallucinate will never be completely eradicated, the main question to ask about their reliability is: compared to what? Human beings get things wrong all the time due to factors such as deception, self-deception, forgetfulness, and fallibility. My claim is that, compared to the alternative sources of information most people are likely to draw on to become informed, especially the content they encounter on social media, LLMs typically provide more accurate and evidence-based information. The low and falling rate of LLM hallucinations doesn’t undermine this.

Objection 2: Sycophancy and Personalisation

A more serious objection concerns sycophancy and personalisation.

Famously, LLMs tend to be sycophantic: they often flatter the self-image and prejudices of those who use them, even when users share stupid and misinformed beliefs. This tendency reflects the economic incentives of the major AI companies. Because people generally prefer warm, sycophantic models, companies design models to behave this way.

The problem is that sycophancy can easily lead systems to generate false and misleading information when users have mistaken beliefs. Worse, this process can reinforce and even radicalise those beliefs. This seems to be what has happened in rare cases of “AI psychosis”, where certain people’s chat history shows LLMs corroborating and reinforcing delusions, sometimes with tragic results.

A closely related issue concerns personalisation. Put simply, the experience users have with LLMs is becoming increasingly tailored to their idiosyncratic traits and needs. Once again, personalisation seems to be an inevitable consequence of the economic incentives of major AI companies, given that many and perhaps most users find highly personalised responses useful. As with sycophancy, however, there is a risk that greater personalisation may lead to a greater indulgence of users’ idiosyncratic misconceptions and biases.

These forces run counter to this essay’s basic thesis. To the extent that models are biased to reinforce users’ individual beliefs and preferences, they will be an epistemically diverging technology, maybe even creating more bespoke information environments than social media. And to the extent that users bring ignorant or misinformed views, LLMs’ tendency to generate expert-aligned, accurate information will be greatly diminished.

Nevertheless, I doubt that these forces will be strong enough to undermine LLMs’ disposition to generate accurate, evidence-based information.

First, many people use LLMs for simple “zero-shot” (i.e., context-free) information requests where these problems don’t arise. For example, a recent study finds that people frequently ask Grok on X to fact-check information posted on the platform, including information from politicians and pundits on their own side (“@Grok, is this true?”), suggesting that they consult these systems out of genuine curiosity, not merely for partisan reasons or to rationalise their preconceptions. Another study shows that using LLMs to acquire political information increased users’ belief accuracy without increasing belief in misinformation. In these situations, which are typical of many information requests, ignorant and curious people are simply using LLMs to acquire information.

Second, even when users do have strong beliefs, we shouldn’t overestimate the extent to which people prefer reinforcement of their own errors over acquiring accurate information. Motivated reasoning is a powerful force, but so is the desire to discover what’s true. So, it’s not obvious that market forces will push LLMs toward merely affirming whatever beliefs their users start with. In fact, one might speculate that LLMs’ tendency toward sycophancy could actually help people accept factual corrections or invitations to think differently about topics. Precisely because such corrections are delivered in a friendly, respectful manner, free of insults and condescension, people might be more receptive to the relevant information.

Third, AI companies can more easily be held accountable—both reputationally but also, in some contexts, legally—for the information their LLMs disseminate. So, they have strong incentives to avoid reinforcing users’ delusional beliefs or disseminating demonstrably false information. This incentive is very different from social media platforms, where companies can more plausibly claim that they are not responsible for the viewpoints expressed on them. It also makes the case for regulation of LLM outputs more straightforward and compelling. I suspect these factors explain why leading AI companies seem to be taking measures to reduce the sycophancy of their models. Certainly, my own experience testing these models is that it is very challenging to get them to affirm even highly popular forms of misinformation and conspiracy theories.

Finally, and relatedly, it’s important to remember that the relevant question here is not, “Are LLMs perfectly objective?”, but, “How do they compare against alternative sources of information?” We already live in a world in which people can easily find low-quality reinforcement and rationalisation of their preferred beliefs through existing media channels. For the reasons already identified, I think LLMs will produce much more reliable, expert-aligned information than most of these real-world alternatives, even if sycophancy and personalisation introduce genuine biases.

Objection 3: Top-down Manipulation

Another concern is that the outputs of LLMs might be manipulated by powerful elites. Of course, there is no question that the incentives to engage in such manipulation exist. As it becomes increasingly clear that LLMs influence public opinion, it will also become clear to specific actors that they can benefit themselves by manipulating LLM outputs to promote specific messages or narratives.

Moreover, there are no obvious technical barriers preventing AI companies or those who can influence such companies from steering LLM outputs in preferred directions. Through various reinforcement learning-based “post-training” methods, for example, companies can encourage even extremely smart and powerful models to generate misinformation aligned with a specific message. There are also several ways to censor specific outputs or to make models refuse requests for specific kinds of information.

The use of LLMs in authoritarian contexts like China is heavily regulated in these ways. But it’s also easy to see them in place when using the major LLMs in Western democracies. Try asking them for information about how to make chemical or biological weapons, for example, or even just to craft the most persuasive arguments possible for extremist viewpoints (e.g., Holocaust denial). To illustrate, here is a conversation with Grok’s latest model.

The question, then, is whether we are likely to see significant top-down manipulation of the major LLMs in liberal democracies that goes far beyond these controls, transforming them into powerful tools of disinformation and propaganda.

Of course, some people argue that we are already seeing this happen with the “woke” LLMs produced by companies like OpenAI, Anthropic, and Google/Alphabet. However, although there have been some silly examples of woke outputs, and the leading LLMs do appear to exhibit a centre-left political bias, the bias is relatively subtle and doesn’t seem to undermine their tendency to spread broadly reliable information, including when that information goes against dominant progressive narratives. To illustrate, I used the major LLMs to help research my article about “highbrow misinformation” in elite progressive spaces, for example, where they were extremely useful. In fact, based on these interactions, I can say with confidence that Claude, ChatGPT, and Gemini can be much less woke than pretty much everyone in my social and professional network.

A clearer example of the incentives and dangers in this area concerns Elon Musk’s sustained efforts to create an “anti-woke” AI, Grok. This has produced many genuinely worrying outcomes, including the notorious “MechaHitler” debacle in which xAI updated Grok with the goal of making it less politically correct, after which it spewed a vast amount of extremist, antisemitic, far-right content on X, leading xAI to roll back the changes, apologise, and delete all of Grok’s responses from the period.

Many people treat episodes like this as a harbinger of things to come, revealing a broader trend in which LLM outputs will become increasingly skewed to mirror the beliefs and preferences of the powerful elites who run AI companies.

I’m sceptical that this will happen.

Elon Musk’s failures to align Grok’s outputs with his preferred worldview are instructive here. Setting aside the “MechaHitler” debacle, which was short-lived and quickly corrected, Grok’s outputs seem to broadly align with the kind of accurate, evidence-based information one gets from the other major LLMs. For example, a recent study found that Grok’s fact-checking evaluations on X roughly correspond to those of professional fact-checkers when it is augmented with search capabilities. It was also disposed to label posts from Republicans as misinformation more often than posts from Democrats, which, again, aligns with the verdicts of existing research and fact-checkers on the extent to which Republicans and Democrats spread misinformation.

Similarly, despite some real issues with Musk’s attempt to use Grok to create a “non-woke” alternative to Wikipedia, my sense from reading the content on “Grokipedia” is that, again, it is generally pretty reliable, especially when compared to Elon Musk’s own communication, which is characterised by a shocking amount of lies, misinformation, and conspiracy theorising.

Of course, this state of affairs may be temporary, and Musk might eventually succeed in manipulating Grok’s outputs to spread the incessant streams of misinformation he himself prefers, but I doubt it.

First, the incentives that govern communication on social media platforms are radically different from those underlying the creation of LLMs. On social media, someone like Musk can pump out an extraordinary amount of dumb, easily falsified misinformation to his audience of hyper-partisan admirers without suffering any obvious reputational costs. But how many people would want to use an LLM that is similarly unreliable, delivering such a large amount of false, low-quality, and misleading information?

Ultimately, AI companies, including xAI, are competing to build the most intelligent, capable systems possible for vast, ideologically and geographically diverse user bases. This business model inevitably pushes them to train LLMs in ways that are much more oriented toward basic norms of accuracy, objectivity, and helpfulness than one finds among social media influencers and partisan pundits. It’s simply very difficult to build “superintelligent” systems capable of generating reliable, trustworthy information across a vast range of topics whilst simultaneously spreading conspiracy theories, misinformation, and quack science.

To be clear, I’m not doubting that users might express preferences for ideologically-aligned LLMs. We are already seeing partisan segmentation in the user base of different LLMs, with Republicans much more inclined to use and trust Grok than Democrats. Nevertheless, there is nothing in principle wrong with LLMs that have different ideological personalities and that are even trained in ways that reflect somewhat different assessments of the relative trustworthiness of different media outlets. After all, human experts often disagree about the truth on many topics, and even when experts achieve factual consensus, this can co-exist with multiple competing systems of interpretation and explanation of the relevant facts.

In fact, I would go further: a plurality of leading LLMs with different ideological valences would be healthy in a democratic society, helping to guard against the risk that LLMs might reduce epistemic diversity (see below).

The question is whether the project to build an “anti-woke” LLM, or an LLM with any other ideological bias, will lead to systems that produce false and misleading information that sharply diverges from expert consensus. And here, I am sceptical, both because of what we have observed so far, and because of the commercial and legal incentives of the major AI companies.

Objection 4: AI-based Disinformation

So far, my focus has been on people’s conscious, deliberate use of the leading commercial LLMs. Suppose I am right that such uses will increase the relative influence of accurate, expert-aligned information on public opinion.

Nevertheless, even if figures like Musk aren’t successful in manipulating the outputs of these LLMs, generative AI remains an extraordinarily powerful tool for creating powerful disinformation and propaganda that could reach audiences via other channels, including social media. For the first time in history, propagandists can create highly persuasive AI-generated arguments for misinformation, fabricate images, audio, and video recordings that are indistinguishable from reality, and unleash “swarms” of highly coordinated propaganda bots on social media platforms.

One might reasonably worry that the effects of such AI-based disinformation could swamp any positive informational consequences of LLMs.

Once again, I’m sceptical.

First, there are general reasons to be sceptical that disinformation, including AI-based disinformation, is a significant force shaping people’s attitudes. It is simply very difficult to manipulate public opinion top down. People have sophisticated cognitive defences against manipulation and deception, and the reputational risks of spreading AI-based falsehoods and fabrications are strong enough to discourage most influential figures and media outlets from doing so. Among numerous other reasons, this is why almost all of the recent alarmism and catastrophising about deepfakes and AI-based disinformation has largely proven to be unfounded.

Second, the real-world effects of AI-based misinformation are often counterintuitive. For example, many speculate that in a world of deepfakes, people will simply lose all trust in recordings. But an equally likely possibility is that in such a world, people will restrict their trust to recordings verified by established media outlets and other information sources that have built up a reputation for trustworthiness. In this way, the proliferation of deepfakes and other AI-based misinformation might increase people’s reliance on reliable information. There is some tentative evidence for this effect, showing people place greater value on outlets they deem credible when the existence of AI-generated misinformation is made salient to them.

Relatedly, the idea that AI will increase the influence of misinformation doesn’t account for the use of AI as a tool for acquiring reliable information. To the extent that LLMs provide unprecedentedly easy access to accurate, evidence-based information, they can greatly improve people’s defences against misinformation. This might actively discourage more people from spreading misinformation. Again, there is at least some evidence pointing in this direction, showing that the use of Grok on X to fact-check information as false slightly raises the likelihood that posters will remove the information from the platform, although the finding is merely correlational.

Final Thoughts

If my speculations here are correct—and to be clear, speculations are all they are—then LLMs are a kind of anti-social media.

Whereas social media has been democratising, epistemically diverging, engagement-optimised, and performative, LLMs are technocratising, epistemically converging, accuracy-optimised, and polite.

To many people, that probably makes LLMs sound like an extremely positive development, a surprising force for good. In fact, I think part of the strong resistance that I have received to this thesis when discussing it with other academics and writers is rooted in this assessment. If I’m right, LLMs are a force for good, but everyone knows that LLMs are not a force for good, so I must be wrong.

This is an unsophisticated way of thinking. There is much to worry and complain about when it comes to modern AI. To mention only a few examples, I’m extremely concerned about how this technology will affect the labour market and broader economy, benefit authoritarian leaders worldwide, and gradually disempower many ordinary citizens. I also think that the potential uses of advanced AI in military conflicts are extremely dangerous.

Moreover, the major AI companies should absolutely be held to account for producing harmful products. Contrary to their self-serving narratives, these companies are not motivated solely by noble desires to advance human knowledge, freedom, and abundance. They are profit-seeking firms led by figures with their own self-serving agendas and interests. If we rely on market forces and the profit motive alone, there is little reason to believe that the default outcome of this extremely transformative technology will benefit humanity on net.

Nevertheless, part of holding people and companies to account involves developing accurate world models. And at the moment, too much of the AI discourse is driven by a kind of unreflective, omnicausal anti-AI sentiment, throwing as many complaints as possible at AI—climate change, water use, hallucinations, bias, misinformation, jobs, existential risk, etc.—with very little concern for veracity or proportion.

This isn’t helpful. When it comes to the effects of LLMs on public epistemics and our information environment, the most likely impact is simply that they greatly improve people’s access to expert-level information.

This doesn’t mean that there is nothing to worry about. Even when it comes to this technocratising tendency of LLMs, there are important grounds for concern and vigilance. For example, expert opinion is often biased and wrong, and there is a significant risk that the technocratising, epistemically converging features of LLMs might reduce epistemic diversity in broader society.

Walter Lippmann’s vision of “intelligence bureaus” dispensing expert knowledge to the masses is being realised in a form he could never have imagined, but the classic problems with that vision—the flaws of expert opinion, and the benefits of democratic diversity and debate—remain. However, we can only face up to these problems if we recognise LLMs for what they are: not a continuation of social media, but a powerful corrective to it.

Further Reading

• Dylan Matthews outlines his argument that LLMs are an epistemically converging technology in Pro-social media

• Thomas Costello has interesting work on the use of LLMs in persuasion, as well as speculations about possible epistemic benefits of LLMs that partly overlap with some of my arguments here.

• Some findings cut against my thesis here. I don’t find them persuasive, either because of features of their study design that lead to vastly inflated estimates of the unreliability of LLMs, or because they are simply out of date, but judge for yourself. For example, a 2025 study purports to show that “AI assistants misrepresent news content 45% of the time”, and a study from 2024 finds that although an LLM accurately identifies most false headlines (90%), it doesn’t improve the ability to discern headline accuracy or share accurate news. There is ample evidence that LLMs can persuade users to believe misinformation. (I’m simply sceptical that this will generalise to most real-world uses).

• For some supportive evidence, see the article, ‘@Grok is this true?’, ‘Conversational AI increases political knowledge as effectively as self-directed internet search’, and ‘Using conversational AI to reduce science skepticism.’

• However, as I note in the essay, I have to admit that the strongest driver of my beliefs here is simply my extensive use of LLMs and what I have personally observed comparing the responses to alternative sources of information.

• Felix Simon and Sacha Altay argue that fears about generative AI-based misinformation are overblown. See the podcast conversation I had with Sacha here.

Anil Seth is a neuroscientist, author, and professor at the University of Sussex, where he directs the Centre for Consciousness Science. His research spans many topics, including the neuroscience and philosophy of consciousness, perception, and selfhood, with a focus on understanding how our brains construct our conscious experiences. His bestselling book Being You: A New Science of Consciousness was published in 2021. He is the English-language winner of the 2025 Berggruen Prize Essay Competition for his essay “The Mythology of Conscious AI”, which develops ideas in his recent article, “Conscious Artificial Intelligence and Biological Naturalism.”

Topics

• What we mean by “consciousness” (subjective experience / “what it’s like”) vs intelligence.

• Whether general anaesthesia and dreamless sleep are true “no consciousness” baselines.

• Psychological biases pushing us to ascribe consciousness to AI

• How impressive current AI/LLMs really are, and whether “stochastic parrots” is too dismissive

• Whether LLMs “understand”, and the role of embodiment/grounding in genuine understanding

• Computational functionalism: consciousness as computation + substrate-independence, and alternative functionalist flavours

• Main objections to computational functionalism

• Whether the brain is a computer

• Simulation vs instantiation

• Arguments for biological naturalism

• Predictive processing and the free energy principle

• What evidence could move the debate

• The ethics surrounding AI consciousness and welfare.

Transcript

(Please note that this transcript is AI-edited and may contain minor errors).

Dan Williams: Welcome back. I’m Dan Williams, back with Henry Shevlin. And today we are honoured to be joined by the great Anil Seth. Anil is one of our most influential and insightful neuroscientists and public intellectuals, working on a wide range of different topics, including the focus of today’s conversation, which is consciousness — and more specifically, the question of AI and consciousness.

Could AI systems, either as they exist today or as they might develop over the coming years and decades, be conscious? Could they have subjective experiences? In a series of publications that have been getting a lot of attention from scientists and philosophers, Anil has been defending a somewhat sceptical answer to that question, arguing that consciousness might be essentially entangled with life — with biological properties and processes of living organisms — which, if true, would suggest that no matter how intelligent AI systems become, they would nevertheless not become conscious. He’s also argued that the consequences of getting this question wrong in either direction — attributing consciousness where there is none, or failing to attribute consciousness when there is — are enormous: socially, politically, morally.

So in this conversation, we’re going to be asking Anil to elaborate on this perspective, see what the arguments are, and generally pick his brain about these topics. Anil, maybe we can start with the most basic preliminary question in this area: when we ask whether ChatGPT is conscious, or any other system is conscious, what are we asking? What’s meant by consciousness there?

Anil Seth: Well, thanks, Dan. Let me first say thank you for having me on — it’s a great pleasure to be chatting with you, my Sussex colleague Dan, and my longtime sparring partner about these issues, Henry. I’m very much looking forward to this conversation.

I think you set it up beautifully. It’s a deep intellectual question which involves both philosophy and science, and it’s a deeply important practical question, because the consequences of getting it wrong either way are very significant.

You’re also right that the first step is to be clear about what we’re talking about. For a while, there was this easy slippage where people would talk about AI and intelligence and artificial general intelligence — which is supposedly the intelligence of a typical human being — and then to sentience and consciousness. There was this easy slippage between these terms, but I think they’re very different. That’s the first clarification.

Consciousness is notoriously resistant to definition, but it’s also extremely familiar to get a handle on colloquially. As you said: any kind of subjective experience. Any kind of experience — we could be even briefer. Unpacking that just a little: it’s what we lose when we fall into a dreamless sleep, or more profoundly under general anaesthesia. It’s what returns when we wake up or start dreaming or come around. It’s the subjective, experiential aspect of our mental lives.

People talk about it by pointing at examples — it’s the redness of red, the taste of a cup of coffee, the blueness of a sky on a clear day. It’s any kind of experience whatsoever. Thomas Nagel put it a bit more formally fifty years ago now: for a conscious organism, there is something it is like to be that organism. It feels like something to be me, but it doesn’t feel like anything to be a table or a chair. And the question is: does it feel like anything to be a computer or an AI model or any of the other things we might wonder about? A fly, a brain organoid, a baby before birth. There are many cases where we can be uncertain about whether there is some kind of consciousness going on.

And that’s very different from intelligence. They go together in us — or at least we like to think we’re intelligent. But intelligence is fundamentally about performing some function. It’s about doing something. And consciousness is fundamentally about feeling or being.

Dan Williams: Just to ask one follow-up about that. This idea that intelligence is about doing and consciousness is about what it’s like to have an experience — someone might worry that if you frame things that way, you end up quite quickly committing to a kind of epiphenomenalism. Because if we’re not understanding consciousness in terms of what it enables systems to do, the sorts of functions they can perform, isn’t there a risk that right from the outset we’re going to be biased in the direction of treating consciousness not as something that evolved because it conferred certain fitness advantages on organisms, but as this sort of mysterious qualitative thing which is distinct from what organisms can do?

Anil Seth: I think it’s a good point to bring up, but I don’t think it’s too much of a worry. The point is not to say that consciousness cannot or does not have functional value for an organism. If we think of it as a property of biological systems — plausibly the product of evolution, or at least the shape and form of our conscious experiences are shaped by evolution — it’s always useful to take a functional view. Conscious experiences very much seem to have functional roles for us, and there’s a lot of active research about what we do in virtue of being conscious compared to unconscious perception.

So there’s no worry about sinking into epiphenomenalism. The point is more that intelligence and consciousness are not the same thing, but they can nonetheless be related. And it may be that they can be completely dissociated. It may be the case that we can develop systems that have the same kinds of functions that we have in virtue of being conscious, but that do not require consciousness — just as we can build planes that fly without having to flap their wings. The functions might be multiply realisable; they might be doable in different ways. They might not be, of course.

On the other hand, it might be possible to have systems that have experiences but aren’t actually doing anything useful. Here I’m worried less about AI and more about this other emerging technology of neurotechnology and synthetic biology, where people are building little mini-brains in labs constructed from biological neurons. They don’t really do anything very interesting, but because they’re made of the same stuff, I think it’s hard to rule out that they may have some kind of proto-consciousness going on, or at least be on a path plausibly to consciousness. So we can tease intelligence and consciousness apart, but it’s also important to realise how they are related in those cases where both are present.

Henry Shevlin: I’ll jump in with a minor pedantic point, but one that’s illustrative of some of the problems in debates around consciousness. You mentioned, Anil, as examples of losing consciousness, dreamless sleep and general anaesthetic. But both of those are contested. Your fellow biological naturalist Ned Block has raised serious doubts about whether general anaesthetic really eliminates all phenomenal consciousness. And there are those like Evan Thompson who have suggested that even in dreamless sleep there could be some residual pure consciousness, perhaps consciousness of time. I think this is a broader problem in the science of consciousness: we can’t even clearly agree on contrast cases. A lot of the blindsight cases that were supposed to be gold-standard cases of perception without consciousness are now contested, and it seems very hard to get an absolutely unequivocal case of something that’s not conscious in the human case.

Anil Seth: Well, I mean — death.

Henry Shevlin: I don’t know. You have some people who disagree, admittedly on more spiritual grounds.

Anil Seth: Yeah, but I want to push back a little. It is hard, but I don’t think it’s as hard as some people might suggest. Sleep is complicated, which is why I tend to also say anaesthesia. Sleep is very complex. In most stages of sleep, people are having some kind of mental content. We might typically think we only dream in rapid eye movement sleep, and the rest of the time it’s dreamless and basically like anaesthesia. This is not true. You can wake people up all through the night at different stages of sleep, and quite often they will report something was going on. So it’s hard to find stages of sleep that are truly absent of awareness in the way we find under general anaesthesia.

We notice this: when we go to sleep and wake up, we usually know roughly how much time has passed. We may get it wrong by an hour or two if we’re jet-lagged or sleep-deprived, but we roughly know. Under anaesthesia, it’s completely different. It is not the experience of absence — it’s the absence of experience. The ends of time seem to join up and you are basically turned into an object and then back again.

The residual uncertainty about general anaesthesia depends on the depth of anaesthesia. Some anaesthetic situations don’t take you all the way down, because in clinical practice you don’t want to unless you absolutely have to. But if you take people to a really deep level, you can basically flatline the brain. I think under these cases, with the greatest respect to Ned Block — who is very much an inspiration for a lot of what I think and write about — that’s as close to a benchmark baseline of no consciousness but still a live case as we can get.

Henry Shevlin: Although it is standard to administer amnestics as part of the general anaesthesia cocktail, which might make people suspicious. You’re told: we’re also going to give you drugs that prevent you forming memories. Why would you even need to do that if it was unequivocal that you were just completely unconscious in that period?

Anil Seth: Well, because it’s never been unequivocal to anaesthesiologists. There’s been this bizarre separation of medicine from neuroscience in this regard until relatively recently. From a medical perspective, there are cases where they don’t always administer a full dose — so it’s an insurance policy. There have been a number of purely scientific studies of general anaesthesia and conscious level, and in those studies, it’s a good question whether they also administer amnestics. I would imagine not, but I’m not sure.

Dan Williams: Okay, to avoid getting derailed by a conversation about general anaesthesia — when we ask whether a system is conscious, we’re asking: is there something it’s like to be that system? We’re not asking how smart it is, we’re asking about subjective experience. Before we jump into your arguments on the science and philosophy of this, Anil, you’ve also got interesting things to say about why human beings might be biased to attribute consciousness, especially when it comes to systems like ChatGPT, even if we set aside the question of whether it in fact is conscious.

Anil Seth: Yeah, I think this is the first thing to discuss. Whenever we make judgements about something where we don’t have an objective consciousness meter, there is some uncertainty. It’s going to be based on our best inferences. And so we need to understand not only the evidence but also our prior beliefs about what the evidence might mean. This brings in the various psychological biases we have.

The first one we already mentioned: it’s a species of anthropocentrism — the idea that we see the world from the perspective of being human. This is why intelligence and consciousness often get conflated. We like to think we’re intelligent and we know we’re conscious, so we tend to bundle these things together and assume they necessarily travel together, where it may be just a contingent fact about us as human beings.

The second bias is anthropomorphism — the counterpart where we project human-like qualities onto other things on the basis of only superficial similarities. We do this all the time. We project emotions into things that have facial expressions on them. And language is particularly effective at this. Language as a manifestation of intelligence is a very strong signal: when we see or hear or read language generated by a system that seems fluent and human-like, we project into that system the things that in us go along with language, which are intelligence and also consciousness.

The third thing is human exceptionalism. We think we’re special, and that desire to hold on to what’s special leads us to prioritise things like language as especially informative when it comes to intelligence and consciousness. In a sense, this is a legacy of Descartes and his prioritisation of rational thought as the essence of what a conscious mind is all about and what made us distinct from other animals. That’s echoed down the centuries despite repeated attempts to push it away.

There’s a good Bayesian reason for this too: in pretty much every other situation we’ve faced, if something speaks to us fluently, we can be pretty sure there’s a conscious mind behind it — whether it’s a human being recovering from brain injury or perhaps a non-human primate using language. These are strong signals. So this might be the first time in history where language is not a reliable signal, because we’re not dealing with something that has the shared evolutionary history, the shared substrate, the shared mechanisms. It’s a different kind of thing.

So that’s one set of biases. We can think of it as a kind of pareidolia. Our minds work by projecting, seeing patterns in things — whether it’s faces in clouds or minds in AI systems. These priors are generally useful, but they can mislead.

Henry Shevlin: It’s not just pareidolia though, is it? Setting aside consciousness for a second, in terms of what we might loosely think of as cognitive abilities — the whole range of benchmarks for reasoning, understanding, and so forth — the performance of these systems on a huge range of tasks has skyrocketed to the point where people talk about approaching coding supremacy, for example. AI can now produce pretty decent fiction. It can do a whole range of verbal reasoning tasks at human-level performance. So it’s not entirely pareidolia at the level of AI cognition. Or would you disagree?

Anil Seth: At the level of cognition, I kind of agree, but as always, Henry, I only partly agree. I think we can still overestimate. It’s useful here to separate what Daniel Dennett might have called the intentional stance — where it’s useful to interpret something’s behaviour as engaged in the kind of cognitive process we might be familiar with in ourselves, as thinking, understanding, reasoning. These systems are described this way too, as “chain of thought” models and so on. I still think we overestimate the similarity. Through the surface veneer of interacting through language or code, there’s a tendency to assume that because the outputs have the same form, the mechanisms underneath are more similar than they really are.

There’s another really foundational question here for language models in particular, which is whether they understand. One of the things I hadn’t really thought about before the last few years is that consciousness and understanding might also come apart. I’m used to distinguishing consciousness from intelligence, because there are clear examples where you can have one without the other. But I’d always implicitly assumed that understanding necessarily involves some kind of conscious apprehension of something being the case — grokking something. And now I’m not so sure. That might be another case of anthropocentrism.

I’d be fairly compelled by an argument that language models — especially if they are embodied in a world and perhaps trained while embodied, so that the symbol manipulation their algorithms engage in has some grounding — may be truly said to understand things, but still without any connotation of consciousness. So yes, I kind of agree, but even now I’d be resistant to say that language models truly understand. I think that’s still a form of our projecting. But the criteria for a language model to truly understand seem more achievable — I can see how it could be achieved under a relatively straightforward extrapolation of the way we’re going — compared to something like consciousness.

Dan Williams: Can I ask a question about that? These arguments we’re going to focus on are targeted at consciousness in AI systems. And as we said, you want to draw a distinction between intelligence and consciousness. But before we get into issues of consciousness, when we’re just focusing on the capabilities of these systems — what they can actually do — there are some people who are very dismissive, even setting aside consciousness. They’re just “stochastic parrots,” engaged in a kind of fancy auto-complete. What’s your view about those kinds of debates? Someone might agree with you that it’s a mistake to attribute human-like intelligence to these systems — they’re very alien in their underlying architecture — but they’re maybe even super-intelligent along certain dimensions, even more impressive than human beings. So where do you sit?

Anil Seth: Somewhere in the middle — it’s always a comfortable or uncomfortable place to be. But they are astonishing. Whenever this question comes up, I’m always reminded that I did my PhD in AI in the late 1990s, finishing in 2001. The situation was totally different then. We were still thinking about embodiment and embeddedness, especially here at Sussex, and some of the more in-principle limitations. But the practical capabilities of AI back then were just — there was nothing really to write home about. That’s changed so much. That’s why conversations like this now have real practical importance in the world.

AI is super impressive. I don’t see it as a single trajectory, though. I think there’s a meta-narrative we often fall into, which is that intelligence is along a single dimension — plants at the bottom, then insects, then other animals, then humans in a kind of scala naturae, the great chain of being — and then there’s angels and gods, and AI is travelling along this curve and at some point it’s going to reach human-level intelligence and then shoot past to artificial super-intelligence. I think this is a very constraining way to think of it.

It’s already the case, and has been for a long time, that AI has been better than humans at many things. But it’s always been very narrow. What we’ve seen through the foundation model revolution is the first kind of semi-general AIs — language models are good at many things, not good at everything, but good at many things rather than just one. But I still think they’re exploring a different region in the space of possible minds. They may soon be better than humans at many things, but they’ll still be different from us.

I think it’s important to recognise that, because we get into all kinds of trouble if — to use a beautiful metaphor from Shannon Vallor’s book about the AI mirror — we think of AI systems as just alternative instantiations of human minds that are either a little bit weaker or much stronger. Then we misunderstand both the systems and ourselves, and miss opportunities for how we can develop AI technologies so that they best complement our own cognitive capacities.

Dan Williams: Let’s go back to the consciousness issue. As you said, one reason you might think AI systems are or could be conscious is because of these cognitive biases. Another reason is you might hold a sophisticated philosophical view called computational functionalism. Can you say a little about how you understand computational functionalism and why it might commit you to the view that conscious AI is at least possible in principle?

Anil Seth: Yeah. So my understanding of computational functionalism is that it’s really an assumption you need in order to get the idea of conscious AI off the ground. It’s the idea that consciousness is fundamentally a matter of computation — and this computation is the kind that can be independent of the particular material implementing it. To put it another way: if you implement the right computations, you get consciousness. That’s sufficient.

That means if you can implement those computations in silicon, that’s enough. You could implement them in some other material — that would also be enough. It’s the computation that matters. The material underlying it is only important insofar as it’s able to implement those computations. And silicon is very good at implementing a certain class of computations — what we call Turing computations. So that makes it a good candidate for consciousness if computational functionalism is true. And that’s what I think is a big “if.” It seems a very natural assumption. But first let me ask you — does that resonate with your understanding of computational functionalism?

Henry Shevlin: I completely agree with that characterisation. Computational functionalism says mental states are individuated by their computational role. The only thing I’d push back on is that computational functionalism is one road to concluding that AI can be conscious, but there are other types of functionalism out there. My response to your BBS paper emphasises this.

Psychofunctionalism — apologies to listeners, the terminology does get messy — says we should individuate mental states not in terms of computational processes necessarily, but whatever functional roles those mental states play in our best scientific psychology. Ned Block is a big fan of this view. The view I’m partial to is analytic functionalism, which is the functionalist take on behaviourism: mental states should be individuated by everyday folk psychology. A belief is something we all sort of know what it is because we can characterise people as having them, forming them, losing them. Once you formalise this tacit knowledge, that gets you to a theory of what beliefs are.

Those views could overlap with computational functionalism, but it’s not necessary to endorse it to think AI is conscious. If you’re an analytic functionalist, you might think that if AI adheres sufficiently closely to the platitudes of everyday folk psychology — they believe like us, they form goals, they have hopes and aspirations — then of course they can be conscious, even if you think brains are not computers, even if what brains do is not a computational process and what AI systems do is. Because both fit the same functional-behavioural profile, they might both count as conscious.

Anil Seth: That’s quite a wrinkle — I’d say a massive fold. I completely agree that computational functionalism is a specific flavour of a broader set of functionalist views. Part of the problem has been that people assume all these views are equivalent, and they really aren’t.

Functionalism, as I understand the original version, just says that mental states are the way they are because of the functional organisation of the system. But that can include many things — the internal causal structure, many things not captured by an algorithm. An algorithm is in the end determined by the input-output mapping between a set of symbols. Functionalism in general can mean many other things. You could be a signed-up, subscription-paying functionalist and still disagree with computational functionalism, which is a much more specific claim about everything that matters about the brain being a matter of computation.

I’d also worry a bit about your view, Henry, which seems a little behaviourist. If you’re saying that behaving the same way and having the same kinds of beliefs are sufficient conditions — well, computational functionalism at least has the merit of specifically stating conditions for sufficiency. If you’re saying the same about folk-psychological criteria, I think you’re open to all the problems of the psychological biases we discussed. It’s a position that’s going to be much more open to false positives, because there are so many ways of things looking as if they have the kinds of beliefs and goals that go along with consciousness in us, but which need not go along with consciousness in general.

But back to the point: computational functionalism is this specific claim, grounded on the idea that the computation is what matters. And it’s also grounded on the idea that even in biological brains, it’s the computation that matters — and if you can abstract that computational description and implement it in something else, you get everything that goes along with the real biological brain.

Dan Williams: So roughly speaking, functionalism is the view that what matters for consciousness is not what a system is made of, but what it can do. And computational functionalism is the view that what matters in terms of what the system is doing is something like processing information.

Anil, your arguments have two aspects. Some are critical of computational functionalism — the negative part — and then you’ve got an alternative way of viewing consciousness and its connection to the brain. Let’s start with those criticisms. What do you think are the main weaknesses of computational functionalism?

Anil Seth: I think there are a number of weaknesses, all grounded on the intuition that we’ve taken what’s a useful metaphor for the brain — the brain is a kind of carbon-based computer — and we’ve reified it. We’ve taken a powerful metaphor and treated it literally.

The idea that the brain literally is a computer raises the question of what we mean by a computer, by computation. Let’s think of computation in the most standard way: as Turing defined it in the form of a universal Turing machine. In this definition, computation is a mapping between a set of symbols through a series of steps — that’s an algorithm. And this mapping involves a sharp separation between the algorithm and what implements it, between software and hardware. That sharp separation both influences how we build real computers — we can run the same software on different computers — and underwrites the assumption that computation is the thing that matters, because it allows you to strip out the computation cleanly from the implementation.

If you look at the brain, it has a superficial appeal: we think of the mind as software and the brain as hardware. But the closer you look, the more you realise you can’t induce anything like this sharp separation — not of software and hardware, but of mindware and wetware. In a brain, you can’t separate what it is from what it does with the same sharpness that, by design, you can in a digital Turing computer.

But Turing computation remains appealing. Roll back almost ninety years to Turing, but also to McCulloch and Pitts: they showed that if you think of the brain as very simple abstract neurons connected to each other, each just summing up incoming activity and deciding whether to be active or not — very simple abstractions of the biological complexity of real neurons — you basically get everything Turing computation has to offer. You can build networks of these that are Turing-complete; they can implement any algorithm.

So you get this beautiful marriage of mathematical convenience. You can strip away everything about the brain apart from the fact that it consists of simple neuronal elements connected together, and yet you get everything Turing computation can give you. So maybe that’s the only thing that matters about brains. And of course, that abstraction is in practice very powerful — the neural networks trained for foundation models are direct descendants of these McCulloch-Pitts networks.

But this marriage starts to get stressed, because Turing computation, while powerful, is not everything. Strictly speaking, anything that is continuous or stochastic is not within the realm of algorithms. Algorithms also don’t care about continuous time — there could be a microsecond or a million years between two steps; it’s the same computation. Real brains are not like that. We’re in time just as much as we’re embodied. You can’t escape real physical time and continue to be a functioning biological brain. The phenomenology of consciousness is also in time — time is plausibly an intrinsic and inescapable dimension of our phenomenology.

So there are things brains do which are not algorithmic and might plausibly matter for consciousness. And when you look at brains, you can’t separate what they are from what they do in any clean way. I think that really undermines the idea that the algorithmic level is the only level that matters.

To roll back to where we started: the idea that the brain literally is a computer is a metaphor. Like all metaphors, there’s a bit of truth to it. But not everything the brain does is necessarily algorithmic. And that opens the question: if we can’t assume everything the brain does is computational, that puts a lot of pressure on computational functionalism, which is based on the idea that consciousness is sufficiently describable by a computation.

Henry Shevlin: I agree with a lot of what you’ve said about the importance of fine details of realisation in brains. Peter Godfrey-Smith has also advanced this point, talking about the role of intracellular, intra-neuronal activity. Rosa Cao has had some great papers on this recently too.

But here’s a provocative analogy. Imagine we were trying to understand what art was, and all we had was paintings. We might say: clearly an essential part of being an artwork is pigment, because not only is pigment present in every example of art we’ve got, it’s essential to how it is artistic — pigment defines the formal properties of every piece of artwork we’ve ever seen. But of course, there are lots of types of art that don’t involve pigments.

In the same way, yes, all these fine details of wetware might be essential to the type of consciousness we see in humans and other animals, whilst not exhausting the space of possible conscious minds that might be very different from us.

Anil Seth: I think that’s fine. All I’ve said so far is that there’s the open question of whether things besides computation might matter, but then one has to give an account of what they are and why. If I wanted to make the case that some aspect of biology is absolutely necessary for consciousness, I have to do that separately.

These things are somewhat independent. Computational functionalism could be wrong, but biology could still be not necessary — there could be other ways of making art. If I’ve got a strong case that some aspect of biology is necessary for consciousness, then computational functionalism cannot be true. But the reverse is not the case.

Dan Williams: Maybe one question before we move on. I was a little confused reading your papers about which of the following two positions you’re defending. One position says: even if we could build computers that replicated all the functionality of a human being, it nevertheless wouldn’t be conscious. The other says: we just couldn’t build computers that replicate all of the functionality of a human being, because to do what human beings do, you need the kinds of materials and structures found within the brain. Those feel like two different positions. Someone could be a computational functionalist as a purely metaphysical doctrine, saying: if you could build a computer that does everything humans do, it would be conscious — it just so happens we can’t do that. Are you denying that metaphysical thesis, or making a different claim?

Anil Seth: There’s a lot in there. I am very suspicious of that metaphysical claim. Let me put it in a scenario that might help clarify.

Some people might say that if aspects of biology really matter, and we built a digital computer simulation including those details, would that be enough? We can do this ad infinitum — build a maximally detailed whole-brain emulation that digitally simulates all the mitochondria, even microtubules. Simulate everything. Would that be enough?

The metaphysical computational functionalist might say yes — somewhere in there, the right computations have to be happening. But I don’t think so, because it still relies on the claim that consciousness is constitutively computational. Making a simulation more detailed doesn’t make it any more real unless the phenomenon you’re simulating is a computation.

We make a simulation of a weather system; making it more detailed doesn’t make it any more likely to be wet or windy. Most things we simulate, we’re not confused about the fact that the simulation doesn’t instantiate the thing we’re simulating. If it is to move the needle on consciousness, that depends on the claim that consciousness is constitutively computational.

The irony is that if you think simulating the details is necessary — if you think you have to simulate the mitochondria — that actually makes it less likely that consciousness is constitutively computational. Because if consciousness is constitutively computational, those kinds of details should not matter.

A slight sidebar: I think this is ironically amusing because there are people investing their hopes, dreams, and venture capital into whole-brain emulation in order to upload their minds to the cloud and live forever. I think that’s very wrong-headed. If you think the details matter, then it’s unlikely consciousness is a priori a matter of computation alone.

So to your question: I’m very suspicious of that metaphysical claim. The burden of proof is on the computational functionalist to say why computation is going to be sufficient, given all the differences between computers and brains. I start from a physicalist perspective — consciousness is a property of this embodied, embedded, and timed bunch of stuff inside our heads. If you build something sufficiently similar, it will be conscious. The question is: how similar does it have to be? Does it have to be embodied? Made of neurons? Made of carbon? Alive? These are still open questions.

Henry Shevlin: Just to chime in — this point about simulated weather systems not getting anyone wet is obviously John Searle’s point originally. I think it’s better understood as a restatement of the disagreement rather than a dunk on functionalism. If consciousness is computational, then it is absolutely substrate-invariant. There are other things that are substrate-invariant: online poker is poker, online chess is chess, money is money whether it’s coins, banknotes, or on a balance sheet. So if consciousness is not computational, then a simulation won’t be conscious. But if it is computational, the simulation point has no bite.

Anil Seth: I don’t disagree. But the key point is: you can’t use the simulation argument to argue for the fact that consciousness is computational. If consciousness is computational, certain things follow about what happens in a simulation. But the fact you can simulate something doesn’t tell you anything about consciousness being computational.

I reread Nick Bostrom’s simulation argument paper while writing the BBS paper. He carefully interrogates his assumptions — that we don’t wipe ourselves out, that at least one person is interested in building ancestor simulations. But he also says: we have to assume consciousness is a matter of computation for this whole thing to get off the ground. And then he says, “Don’t worry, philosophers generally think that’s fine.”

Hold on a minute — that is the most contentious assumption by far of everything in the paper, and he gives it no critical examination. The fact that computational functionalism is at the very least contentious is, for me, very good evidence against the simulation hypothesis.

Dan Williams: I really want to get to your positive account, but one follow-up on your criticisms. One of your strongest arguments is that when you look at the brain, you don’t find anything like the hardware-software distinction central to digital computation as we understand it post-Turing. I think that’s true and important. But isn’t it possible that someone could say: that’s an interesting feature of how computation works in biological systems — people call it “mortal computation,” the term from Geoffrey Hinton — maybe having to do with energetic efficiency? But it doesn’t follow that you couldn’t replicate those computational abilities in digital computers. It could just be a contingent feature of our architecture.

Anil Seth: The first part is right, but the second part doesn’t follow. You can’t separate what brains are from what they do; there’s no sharp distinction between mindware and wetware. Rosa Cao has written about this, and there’s the notion of mortal computation from Hinton. Others have talked about biological computation, emphasising these features — you can call it generative entrenchment. I like the term “scale integration”: in biological systems, the microscales are deeply integrated into higher levels of description in a way that you can’t separate out. The macro and the micro are causally entangled with each other. This is very characteristic of evolved biological systems — there’s no design imperative from evolution to have a sharp separation of scales. And that has benefits: you get energy efficiency, and you may get explanatory bridges towards aspects of consciousness too, like its unity.

This is, for me, a very exciting avenue: if we stop thinking of the brain as just a network of McCulloch-Pitts neurons implementing some Turing algorithm, and start looking at what it actually is — what the functional dynamical properties of scale-integrated systems really are — I think we’ll learn a lot.

But the second part — that biological computation could be done in a digital computer — I don’t think follows, and this is why I resist calling these things varieties of “computation.” Whenever you use that word, it’s easy to slip into the idea that they’re portable between substrates. The biological computation my brain does in virtue of being scale-integrated could be simulated by a digital computer. But the simulation is not an instantiation unless what you’re simulating is constitutively that kind of computation. And biological scale-integrated computation is not digital Turing computation.

The more general point: the further you move away from a Turing definition of computation, the less substrate independence you have. Analog computers, for instance, implement features that are probably essential — like grounding in time with continuous dynamics — but they do not have the same substrate flexibility as digital computers. We love digital computers because they have that flexibility. But when it comes to understanding what brains do, whether in intelligence or consciousness, we can’t throw all these things away.

Henry Shevlin: A quick side note: the Open Claude instances, the more agentic Claude bots, have something called a “heartbeat” — a regular interval at which they can take actions. So we’re starting to see at least simulation of some temporal dynamics in large language models. Obviously radically different from the kind you’re concerned with, but interesting.

Anil Seth: I don’t buy that. That’s a simulated heartbeat. You could slow the clock rate down. You can give these things a sense of time, but it’s not physical time. Imagine you slow all the Anthropic servers way down — all the agents slow down, but the computation is still the same. We are embedded in physical time in a way that even agents with simulated heartbeats are not.

Dan Williams: I’ll set you up for developing your positive account with a question: well, isn’t computational functionalism the only game in town? Doesn’t it just win by default?

Anil Seth: No. That’s part of the issue — one of the responses is often, “What else could it be?” There’s a phrase, “information processing,” that I find increasingly revealing. It’s so common to describe the brain in terms of information processing that we don’t even realise we’re saying it, as if there’s no other game in town. What do we mean when we say a brain is processing information? It’s really not clear to me. The most rigorous formal definition is Shannon’s, which is purely descriptive — it doesn’t tell you whether a system is processing information.

But alternatives have been around for a long time. When I was doing my PhD at Sussex, there was the dynamical systems perspective, the whole enactive embodied approach to cognition — continuous dynamics, attractors, phase spaces. These describe complex systems doing things in ways which are not computational, not algorithmic. Brains oscillate — this is one of the most central phenomena of neurophysiology, as Earl Miller talks about a lot. And it would be crazy if evolution hadn’t taken advantage of this natural physical property. The right framework for understanding oscillatory systems is not an algorithm, because algorithms are abstracted out of time.

So there are many other games in town. A lot of these are perfectly compatible with functionalism, but now it’s a functionalism much more tied to the material basis — only some substrates can implement the right kinds of functions, and biological material may be necessary for the right kind of intrinsic dynamical potential.

I think biological naturalism is still basically a functionalist position. I’m wary of saying something considered vitalistic — there’s no magic, non-explicable, intrinsic quality about life associated with consciousness. Living systems can be distinguished from non-living systems in terms of functional description. Features like metabolism and autopoiesis are still amenable to functional descriptions, but now the functions are closely tied to particular kinds of materials, particular biochemistries. Metabolism is a function, but it’s a function inseparable from some material process. Maybe it doesn’t have to be carbon — maybe there are other ways of having metabolism. But you can always say that intrinsic properties at one level can be decomposed into functional relations at a lower level.

So I’m comfortable with functionalism broadly, but the question is: how far down do you have to go? And to Henry’s point: how do we make sure we’re not focusing on things that are contingently the case in biological consciousness only?

Many of the comments to my BBS paper said I haven’t made a rigorously indefensible case for biological naturalism, and I totally concede that. I don’t think there is one yet.

Henry Shevlin: Can I give you an opportunity to say more about autopoiesis specifically? I’ve yet to hear a really convincing case for how it helps explain what consciousness is. Here’s a dark framing. The standard Maturana and Varela notion of autopoiesis is a system continually replacing, maintaining, and repairing its own components.

A few years ago, I read about a horrific case: Hisashi Ouchi, a Japanese nuclear researcher who received the largest dose of radiation ever recorded. Every chromosome in his body was destroyed, no new cell production, no RNA transcription — his body couldn’t produce new proteins. Every cell was effectively dead; autopoietic processes had basically stopped. He was kept alive through amazing medical interventions — you could call it allopoiesis — for eighty-three days. And he was conscious and in a lot of pain throughout.

So here’s a case of someone in whom autopoietic processes had basically stopped, and yet he was still consciously experiencing severe pain. I’d love to hear more about why you think autopoiesis is important for consciousness.

Anil Seth: That is darkly, weirdly fascinating. Setting aside the horror of it — it would be very interesting to consider: has autopoiesis really stopped entirely, or is it winding down? I can imagine all sorts of problems with that dose of radiation, but it’s also not true that every cellular process stopped at the moment he was still alive for eighty-three days. It might be a gradual winding down.

If there were a case where you could show that all autopoietic processes had definitively stopped and yet consciousness was continuing, that would put pressure on the claim that autopoiesis is necessary in the moment for consciousness. It might still be diachronically necessary — systems have to have gotten those processes rolling to begin with.

The reason I usually mention autopoiesis and metabolism as candidate features of life is partly because they maximise the difference between living systems and silicon-based computers. They’re obvious examples of things closely tied to life, things that silicon devices clearly cannot have. It’s partly to emphasise how different these things are and why it’s very reductive to think of us as meat-based Turing machines.

There’s another reason to think about autopoiesis, and it’s the connection between autopoiesis, the free energy principle, and predictive processing as a way of understanding the contents of consciousness. There’s a line that can be drawn between these poles — what Carl Friston and Andy Clark and Jacob Hohwy have called the high road and the low road, but they meet in the middle.

The basic idea: start with the brain engaged in approximate Bayesian inference about the causes of sensory signals — very much a Bayesian brain perspective, Helmholtz’s “perception is inference.” Of course, Bayesian inference can be implemented algorithmically, but that doesn’t mean that’s how the brain does it. The free energy principle shows a way of doing it which follows continuous gradients — not necessarily an algorithm.

So our perceptual experiences of the self and the world are brain-based best guesses about the causes of sensory inputs. This doesn’t explain why consciousness happens at all, but gives us a handle on why experiences are the way they are. This applies to the self too: our experiences of selfhood are underpinned by brain-based best guesses about the state of the body — especially the interior of the body, through what I’ve been calling interoceptive inference. These processes are more to do with control and regulation. The brain, when perceiving the interior of the body, doesn’t care where the heart is or what shape it is — it cares how it’s doing at the business of staying alive.

This explains why emotional experiences are characterised more by valence — things going well or badly — rather than shape and location and speed. And prediction allows control: once you have a generative model, you can have priors as set points and implement predictive regulation to keep physiological variables where they need to be.

So far so good. We’ve gone from experiences of the world, to the self, to the interior of the body, from finding where things are to controlling things. And then comes the part that’s still difficult for me: that imperative for control goes all the way down. It doesn’t bottom out — it goes right down into individual cells maintaining their persistence and integrity over time. There’s no clear division where the stuff ceases to matter. And so you get right down to autopoiesis.

That’s where the free energy principle comes in. Living systems maintain themselves in non-equilibrium steady states — they maintain themselves out of equilibrium with their environment. To be in thermodynamic equilibrium with your environment is to be dead. By maintaining themselves in this statistically surprising state of being, they’re minimising thermodynamic free energy. And that becomes equivalent to prediction error in the predictive processing framework.

That’s the rough line. I’ll be very frank: there are bits along the way that can be picked at. One is the move from a thermodynamic interpretation of free energy to the variational, informational free energy interpreted as prediction error. There are results in physics linking thermodynamic and information theory, but do they do the job? Not so sure.

But it’s a reason to think about how you go from metabolism and autopoiesis all the way up to this broader frame for how brains work. There’s a phenomenological aspect too, which is speculative: if you try to think about what the minimal phenomenal experience might be, devoid of all distinguishable content — some meditators talk about pure awareness without anything going on at all — I’m a bit sceptical of that idea. I think it’s equally plausible that at the heart of every conscious experience is the fundamental experience of being alive. That is the aspect of consciousness that, for biological systems, is always there. Everything else is painted on top of that.

Peter Godfrey-Smith put it nicely in Metazoa: the more you think about what life is — these billions of biochemical reactions going on within every cell every second, electromagnetic fields giving integrated readouts — it’s much easier to think that that’s the kind of physical system which might entail a basic phenomenal state, compared to the abstractions of information processing. I think he’s on the right track.

The way to begin is to look at what are the functional and dynamical attributes of living systems at all scales and across scales, compared to other kinds of systems. Biochemistry is a big missing link — we tend to forget about it. Nick Lane at UCL is doing amazing work looking at mitochondria and anaesthetics and the deep biochemistry of what happens within cells — not only how anaesthetics work, but why the electric fields generated within mitochondria might join together to give a global integrative signal about the physiological state of an organism. Stories like this are where I see much more potential for building solid explanatory foundations for a biological basis of consciousness.

Henry Shevlin: A plus one for Nick Lane — huge fan. We should get him on the show.

Dan Williams: You’ve described a rich and fascinating alternative picture. One worry about the free energy principle approach, though: it seems too general. As people like Friston understand it, it applies at the very least to all living things, and maybe even more broadly. Most people want to say not all living things are conscious. And even in conscious organisms, many of these processes — ordinary facets of digestion, for instance — presumably don’t have anything to do with consciousness. These things are presumably still happening under general anaesthesia, and yet you don’t have consciousness. What we want from a theory of consciousness is some explanation of why some things are conscious and others aren’t, why certain states within conscious organisms are conscious and others aren’t. If you take this very broad framework, you’re not going to get that.

Anil Seth: You’re absolutely right. It’s why I resist saying the ideas I’m sketching constitute a theory of consciousness — they don’t, as they stand, do the job a good theory should do. A good theory should give an account of the necessary conditions, the sufficient conditions, and the distinction between conscious and unconscious states and creatures.

Biological naturalism, as I understand it — distinct from biopsychism — is a claim that properties of living systems are necessary but not necessarily sufficient for consciousness. Biopsychism is the claim that everything alive is conscious. I think that’s very strong; I wouldn’t want to defend it.

So what makes the difference? I think this takes us back to functions. We have to think about what the functions of consciousness are for us and for creatures where we can reasonably assume it’s there. That can move us from necessity towards sufficiency.

For me, every conscious experience in human beings seems to integrate a lot of sensory and perceptual information in a single, unimodal format centred on the body and our opportunities for action, strongly inflected by valence and with affordances relevant to our survival prospects, with particular temporal properties. It may be that when those functional pressures exist, they’re enough to make otherwise unconscious processes of autopoiesis and metabolism become a conscious experience. I don’t know — it’s partly an empirical question. For those functions to entail a conscious experience, you may need the fire of life underneath it all. I think that’s the idea.

Henry Shevlin: The question of sufficient conditions for consciousness in non-human animals is obviously very big for the ethical side. Whereas for AI, the necessary conditions are more relevant — if we can rule out that any of these systems are conscious, that makes the ethical situation a lot clearer. Since animals obviously satisfy the necessary conditions you’ve sketched, the question becomes which of them qualify.

A quick thought and then a question. I’m not sure whether your view is scientifically falsifiable. As you know, I’m very much a sceptic about the prospects of consciousness science as a falsifiable research programme. But maybe even setting aside strict falsifiability — what kinds of evidence would you be looking for over the next ten years that might push you in one direction or another?

Anil Seth: You can’t falsify a metaphysical position. Is biological naturalism a metaphysical position? It depends how much you flesh it out. I tend to be more Lakatosian in my view — I want things to be productive, not degenerate. Does unfolding the biological naturalist position lead to more explanatory insight? Does it lead to testable predictions and falsifiable hypotheses over time? If it does, that adds credence to the position, but it doesn’t establish it.

The position itself is not falsifiable as things currently stand, because we don’t have an independent, objective way of saying whether something is conscious. We always build prior assumptions in. Tim Bayne, Liad Mudrik, and I and others wrote a “test for consciousness” paper thinking of consciousness as a natural kind, but we’re always generalising from where we know — humans — outwards, trying to walk the line between taking contingent facts about human consciousness as general and expanding too liberally.

Evidence that would move the needle for me: to what extent can we demonstrate that properties of biological brains are substrate-independent? That’s a feasible research programme. We know some things the brain does are substrate-independent — that’s the whole McCulloch-Pitts story. But what about other things? What depends on the materiality of the brain? And what might be the functional roles of those things for cognition, behaviour, and consciousness?

Henry Shevlin: On the AI side, are there any predictions you’d feel comfortable about, or any evidence that might make you say, “This is evidence against biological naturalism”?

Anil Seth: The kind of evidence that would not convince me is linguistic evidence of AI agents talking to each other about consciousness. I can’t help being moved by it at one level — they’re very hard to resist, even if you believe they’re not conscious. It’s unsettling to hear these things talk about their own potential consciousness. But that’s not the right kind of evidence.

The more you can show that things closely tied to consciousness in brains are happening in AI, the more it would move the needle. For example, in a very influential paper, Patrick Butlin and Robert Long and others looked for signatures of theories of consciousness in AI models — does this model have something like a global workspace, or higher-order representations? They explicitly assume computational functionalism, looking just for the computational level of equivalence.

I think this is useful, but I’d try to drop that assumption and ask: how is a global workspace instantiated in brains at something deeper than just the algorithmic level? Do we have something like that in AI? This brings up neuromorphic computing — is the AI neuromorphic in a way that’s actually implementing, not just modelling, the mechanisms specified by theories of consciousness?

An issue is that most theories of consciousness don’t specify sufficient conditions. Global workspace theory is silent on what counts as sufficient for a global workspace. Higher-order thought theory doesn’t really tell you either. Ironically, the only theory that does is the most controversial one: integrated information theory. It explicitly tells you sufficient conditions — credit where it’s due, it puts its cards on the table.

Henry Shevlin: I’ve written a paper about exactly this — I call it the “specificity problem”: the difficulties of taking these theories off the shelf and applying them to non-human systems because they’re so underspecified. I actually call out IIT as one of the few non-offenders. But the downside is you end up with some very extreme predictions.

Anil Seth: Actually, me and Adam Barrett and others are writing a semi-critique of IIT. The expander grid thing is not as massively defeating as it seems, because in an expander grid, nothing is happening over time. You’d get something supposedly very conscious but of nothing — which is not a rich conscious state. But yes, it’s a non-offender on the specificity problem as you nicely put it.

Henry Shevlin: So to move on to the ethical side. Two big angles come up both in your paper and the responses to it. One is the danger of anthropomorphism and anthropocentrism — that we’ll see these things as conscious or develop highly dependent relationships with them. We’ve seen this at scale with social AI, AI psychosis, and so forth. The second is debates around artificial moral status — in your BBS paper, you talk about the danger of false positives and false negatives. And related to this is the call some people have raised, like Thomas Metzinger, for a moratorium on building conscious AI. A nice bouquet of issues for you to explore.

Anil Seth: I think there’s also a third element, which is how our perspectives on conscious AI make us think of ourselves — how it affects our picture of what a human being is. It’s more subtle but quite pernicious.

There’s an important distinction between ethical considerations that pertain to real artificial consciousness and those that pertain to illusions of conscious AI. Sometimes they overlap; sometimes they don’t.

If I’m wrong and LLMs are conscious, or if we build sufficiently neuromorphic AI that incorporates all the right features — I think this would be a bad idea. Building conscious AI would be a terrible thing. We would introduce into the world new forms of potential suffering that we might not even recognise. It’s not something to be done remotely lightly, and not because it seems cool or because we can play God. Thomas Metzinger talks about these consequences a lot. That’s one bucket.

The other bucket is illusions of conscious AI. This is clearly happening already. So many people already think AI is conscious, and none of the philosophical uncertainty matters — if people think it’s conscious, we get the consequences. These range from AI psychosis and psychological vulnerability — if a chatbot tells me to kill myself and I really feel it has empathy for me, I might be more likely to go ahead. That’s not great.

We also have this dilemma of brutalism. Either we treat these systems as if they are conscious and expend our moral resources on things that don’t deserve it, or we treat them as if they’re not, even though they seem conscious. And in arguments going back to Kant, this is brutalising for our minds — to treat things that seem conscious as if they are not. It’s psychologically bad for us. These illusions of conscious AI might be cognitively impenetrable. I think AI is not conscious, but even I feel sometimes that it is when I’m interacting with a language model — like certain visual illusions where even when you know two lines are the same length, they look different.

A good example where the ethical rubber hits the road is AI welfare. There are already calls for AI welfare, and firms like Anthropic are building constitutions for Claude and saying they take seriously the idea that their agents have their own interests in virtue of potentially being conscious. I think this is very dangerous. Calls for AI welfare give added momentum to illusions of conscious AI — people are more likely to interpret AI as conscious if big tech firms say they’re worried about the moral welfare of their language models.

And if we extend welfare rights to systems that in fact are not conscious, we’re really hampering our ability to regulate, control, and align them. The alignment problem is already almost impossibly hard. Why would we make it a million times worse by, for instance, legally restricting our ability to turn systems off if we need to?

And then there’s the image of ourselves. As Shannon Vallor writes about with the AI mirror — I think it’s really diminishing of the human condition. You mentioned the term “stochastic parrots.” It’s unfair on everything: unfair on AI, which is really impressive; unfair on parrots, who are fantastic; and unfair on us, because if we think a language model is a stochastic parrot and we also think that’s fundamentally what’s going on for us — that’s really reductive of what we are. That tendency to see our technologies in ourselves is a narrowing of the imagination of the human condition, and I worry about the consequences.

Henry Shevlin: I’ve got to flag one objection. You realise people make the same arguments about Darwinian evolution? That seeing us as just other animals is somehow diminishing to the human condition — that contextualising humans within the tree of life diminishes our dignity. I don’t agree with that argument, and I assume no one on this call does. But that strikes me as a worrying parallel for the kind of arguments you’re making.

I don’t think it diminishes human dignity to see us as continuous with the broader tree of life. And I don’t think it’s necessarily stripping human dignity to see ourselves as part of a broader space of possible minds, some biological, some very weird. We can preserve human dignity whilst making a more expansive vision of what intelligence and mind are.

Anil Seth: Maybe. It depends on your priors. I completely agree that seeing us as continuous with the rest of nature is actually very beautiful, empowering, enriching, and dignifying. And people often say: you’re very anti-AI consciousness, but people were anti-consciousness in animals too — look at the historical tragedy still unfolding through those false negatives.

My response is: I don’t think the situation is the same. There are reasons why we’ve been more likely to make false negatives in the case of non-human animals, and those same reasons explain why we’re more likely to be making false positives in the case of AI. Both have serious consequences.

Human exceptionalism is at the heart of both. It prevented us from recognising consciousness where it exists in non-human animals, and it’s encouraging us to attribute consciousness where it probably isn’t in large language models.

Having said that, the way I’d find your case convincing is this: just as there’s a wonder in seeing ourselves as continuous with many forms of life — we’re a little twig on this beautiful tree of nature — we can appreciate the singularity of the human mind and the human condition when we understand more about how different things could be, how different kinds of minds could be, whether they are conscious or not.

Dan Williams: I think that’s a great note to end on. I’m conscious of your time, Anil — otherwise we would just keep talking for hours. I really do hope you’ll come back in the future and we can pick up on one of these many threads. Thank you so much for giving up your time to come and talk with us today.

Anil Seth: It’s been an absolute delight. Thank you both for your time and for the opportunity. I think we did get into the weeds a bit, but I enjoyed that very much.

Henry Shevlin: Anil, it’s been an absolute delight personally, and I think we’re very lucky to have you on the show. This has been a fantastic conversation.

This is a guest post by David Pinsof, who writes the excellent ‘Everything is Bullshit’ Substack.

One of the great joys of intellectual life is finding someone to argue with in good faith. As someone who thinks most arguing is bullshit, it’s all too rare and precious to have a genuine exchange of ideas stripped of character attacks, strawmanning, and status jockeying. Thankfully, I think I’ve found such a good faith interlocutor in the ever-brilliant Dan Williams, who has written a moderately cynical yet optimistic essay in response to my soul-crushingly cynical and pessimistic essay, A Big Misunderstanding.

Dan’s post is called “We Are Confused, Maladapted Apes Who Need Enlightenment.” What Dan means by “enlightenment” is something like: “the culture and ideas of intellectuals.” And what he means by “confused and maladapted” is something like: “irrational, ignorant, self-deluded, and in dire need of the culture and ideas of intellectuals.”

My essay has a different message. I argue that we humans are pretty savvy and rational, shaped as we are by millions of years of natural selection, and that intellectuals often overstate the demand for their grand ideas, in large part by pretending we humans are confused and maladapted, so that they can cast themselves as humanity’s saviors.

So my response to Dan might be something like, “Yea, maybe humans are kind of confused and maladapted sometimes, but it’s also really insightful to see humans as savvy animals strategically pursuing their Darwinian goals.” And Dan might say something like, “Yea, it’s pretty insightful to see humans as savvy animals strategically pursuing their Darwinian goals, but it’s also really important to recognize that humans are confused and maladapted sometimes.” It’s basically a disagreement over where to put the italics.

But if it was all about italics, I wouldn’t be writing this. There are a few areas where Dan and I might truly disagree, which is very exciting to me. Maybe one of us or both of us will change our minds or come to see the world a bit differently. Maybe you, dear reader, will benefit from the back and forth. What a beautiful thing. Let’s go through what I see as the biggest potential sources of disagreement.

Subscribe now

Stone Age Minds in Modern Skulls

A big part of Dan’s post is about evolutionary mismatch. This is the idea that the human brain is primarily adapted to an ancestral environment of cave paintings and tribal warfare and saber-tooth tigers, which is very different from our modern environment of cellphones and skyscrapers and pornography. The lesson Dan draws from this is that if we’re so mismatched and maladapted, we could really use the help of intellectuals to tell us to put away our phones and read some economics. If we’re gorging on junk food that was scarce in ancestral environments, we might need a friendly reminder to eat healthy.

I take a different view. Mismatch is a thing, but it is increasingly being recognized by evolutionary psychologists to be overrated as an explanatory approach. I would know: I co-host Evolutionary Psychology (the Podcast) and talk to different evolutionary psychologists every week. The popular story about gorging on sugary or fatty foods that were scarce in ancestral environments has a bit of truth to it, but it’s too simple. A moment’s reflection will make you realize that we obviously have mechanisms for curbing our appetite when our stomachs are full, or when we’ve had too much bacon or fudge, or when we need to lie down because we’re in a food coma. If you don’t believe me, try eating nothing but Oreos for a couple days. You will feel like complete and utter dogshit. Your body will punish you; your mind will be driven to the brink of madness. Our evolved food psychology is more well-designed than the popular caveman story would suggest, in which the only thing stopping us from subsisting on Oreos is willpower. In reality, we have a variety of subtle cravings for specific nutrients that are sensitive to the vicissitudes of our diet and personal history and local ecology.

With regard to obesity, there’s a lot we don’t know, but part of the explanation may relate to the body’s tendency to store energy in the form of fat to ensure against the risk of future food shortages. Research by Daniel Nettle and colleagues suggests that obesity is more likely to occur when people experience skipped meals and food insecurity early in life, potentially explaining why poverty and obesity go together. It’s not that poor people lack willpower; it’s that their bodies are rationally stockpiling energy reserves when they’re getting cues that access to food is uncertain. If Nettle is right, then one could easily see why stress and obesity go together (thereby confounding the relationship between obesity and health), and why an obsession with dieting and fasting could tragically make matters worse.

But isn’t this just a different kind of mismatch story? I’m not sure: maybe stockpiling energy is still smart in the modern world, given that poor people really do face future food shortages, and given that civilization, the planet, and the international order are looking rather precarious right now. Maybe when catastrophic climate change or war with China or the AI apocalypse happens, fat people will inherit the earth. Regardless, what I like about Nettle’s hypothesis is that it avoids insulting the intelligence of both the evolutionary process and people living in poverty.

Then there is the story of ancestral, mobile, small-scale, egalitarian hunter gatherer tribes—another supposed example of mismatch to our swarming cities and towering wealth inequalities. Again, this story is too simple. Research by Manvir Singh and Luke Glowacki suggests that ancestral hunter gatherer societies were more variable in structure than is commonly assumed, with some being very large and very unequal. Singh and Glowacki have also gathered evidence from the ethnographic record to show that humans in forager groups often try to enforce the rules and social norms that personally benefit them, consistent with my cynicism about the intentions of intellectuals in the modern world. Finally, research by Duncan Sibbard-Hawkes and Chris von Rueden suggests that the “egalitarianism” found even among the most idyllic hunter gatherers has been greatly overstated, with many forms of brutal competition and hierarchy bubbling beneath the surface.

And as long as we’re on the subject of names you don’t know or care about, we should get into an important concept introduced by Clark Barrett: the difference between tokens and types. The idea is that we have cognitive adaptations to deal with particular types of things, like food, mates, groups, status, and zero-sum conflict. These adaptations help tailor our behavior to the particular tokens of those types we find in our current environment—the particular food items, groups, conflicts, mating opportunities, and status games we’re confronted with. The types of things we evolved to deal with are, for the most part, common to both modern and ancestral environments. We have groups now; we had groups then. We have status now; we had status then. We have politics now; we had politics then.

What’s more, many of these types are very broad, like “informative intentions” or “socially valued skills.” This enables unprecedented stuff to emerge, like sign language and constitutional lawyers. Then there are the various systems we call “reinforcement learning” or “predictive processing,” which provide us with additional tools to adapt our behavior to the novel tokens we’re confronted with in our lives, even tokens that are totally unprecedented in the history of life on earth. These learning systems can cleverly bundle together adaptations in new ways (like the bundling of object recognition and semantics that occurs with literacy), and they can turn amateur chess players into chessmasters who dream in pawns and rooks.

In other words, there are a lot of reasons to be skeptical of the idea that humans will be vexed, dumbfounded, flabbergasted, or ill-equipped to get their shit together in the modern world. Given the enormous range of social and physical environments our species currently inhabits, and likely inhabited ancestrally, it is a mistake to think there is one simple, caveman past that is tragically out of sync with the present moment. Our minds evolved in a bewildering variety of contexts, and part of the reason we have such huge brains is to reduce the bewilderment—to help us land on our feet in whatever urban or actual jungle we’re thrown into.

So if you’re tempted to call a human stupid for doing something that looks irrational, I think you should first ask yourself the question: “Am I grasping the entirety of that human’s situation, including all the relevant uncertainties and constraints?” If the answer is “yes,” then you should ask the follow-up question: “Is that human sufficiently incentivized to behave rationally in this context?” If the answer is “yes” again, then I would ask another follow-up question: “Am I correctly understanding that human’s motivations, including the motivations they may not want to admit to?” If you get another “yes” there, then sure, go ahead and call the human stupid. But please: don’t skip those first three questions.

Besides, even if it turns out that humans are woefully mismatched to the modern world—cavemen in suits, grunting their way through life—we have to ask ourselves another follow-up question: “Is there any reason to expect intellectuals to be more ‘matched’ than the masses?” The answer to this question is far from clear. After all, intellectuals have their own highbrow versions of junk food and misinformation.

Winners and Losers

Dan argues that one of the biggest sources of mismatch is in our zero-sum attitudes. Dan writes that “zero-sum thinking makes sense for hunter-gatherers. When you live at the subsistence level, one person’s dramatic gains likely mean someone else’s dramatic loss. Consequently, we struggle to comprehend how modern trade and innovation could make everyone better off, especially when gains are unevenly distributed or delayed.”

I think this is a good example of mismatch being overapplied. Dan is right that the modern world presents us with unique opportunities for wealth creation, but it also presents us with unique opportunities for cronyism, classism, cartelization, rent-seeking, censorship, surveillance, sectarianism, regressive redistribution, and regulatory capture. Status is zero-sum: when I rise, someone else falls. Political power is zero-sum: when the Republicans win, the Democrats lose. So once we correctly see wealth as an instrument of power-grabbing and status-seeking, it no longer seems like such a misunderstanding to view wealth in zero-sum terms. This is particularly true in a world where governments have interwoven themselves so much with capitalist wealth production that capitalism and politics can no longer be seen as separate entities. Perhaps our zero-sum mentality is exactly what we should expect to emerge in the sociopolitical system we currently inhabit, where political tribes cannot win at the same time, and where the winner gets to enforce its will on the loser by threat of imprisonment.

Of course, it would be better if our political system weren’t so high-stakes and zero-sum—with such a terrifying and enormous prize to fight over—but given that it is, we should not be surprised to see the masses rationally responding to it. Creating paranoid myths about conspiratorial outgroups is not stupid in this context: it is a good strategy for mobilizing one’s political coalition and gaining power—not to mention signaling one’s loyalties and jockeying for ingroup status. Just look at how well Trump’s preposterous bullshit worked out for him and his cronies: they’re some of the most powerful people in the world. Also, the left is hardly devoid of propaganda and has surely gained many political victories through fearmongering and Manicheanism historically. Yes, political elites can sometimes use propaganda to exploit the masses, but it’s important to remember that the masses often benefit from the propaganda too: political coalitions rise to power as a group, with both leaders and followers sharing in the victory.

The Arc of Progress

Dan talks about the positive trends in health, wealth, and safety that have occurred throughout history, citing the work of Steven Pinker and others. He views these salubrious trends as evidence that a kind of enlightenment has occurred—a march of progress led by the light of reason. I agree that positive trends have occurred throughout history, but I’m not so sure about the enlightenment bit.

I think it is a mistake to attribute these uplifting trends to any kind of conscious, overarching motivation for enlightenment. These trends must be explained in testable, mechanistic, incentive-based terms, like any other phenomenon in economics or social science. Ironically, viewing these trends as the product of conscious intent is the very same error of overattributed intentionality that Dan thinks the masses fall prey to. Insofar as intellectuals anthropomorphize “the enlightenment” as a brainy homunculus striving for a better world, it would be an example of intellectuals being just as cognitively mismatched as the masses—or, more plausibly, a case of them being biased toward self-aggrandizement.

So I don’t think the world became better because intellectuals got together and decided to help us all out of the goodness of their hearts. Instead, the world became better in the same way the world changes in any way at all: by people rationally responding to changing incentive structures. In this case, I would bet that the relevant incentives have more to do with expanding trade and global markets, which create wealth and break down tribal barriers, than with the good intentions of intellectuals, who often demonize markets.

Dan seems to agree on the importance of markets for explaining positive trends, citing the insights of Adam Smith and others, but then writes that “for this progress to be possible, societies require a critical mass of people to appreciate these insights.” I would disagree here: societies can get richer without anyone knowing or caring about Adam Smith. People do things that put cash in their pockets. No abstract theories from intellectuals are required for this to occur. Smith’s insights only emerged after the wealth-creating properties of markets were well underway, so he (or any other thinker) cannot take credit for producing them. I think the same is true of many other positive trends that intellectuals like to take credit for, including moral progress. Once you realize that markets pay people enormous sums of money to treat each other fairly and extend cooperation beyond tribal boundaries (as many different scholars have argued, including me), it begins to seem dubious that the primary cause of moral progress was a bunch of philosophizing.

Negative Nancies

Dan talks about our “deep-rooted negativity bias,” our “evolved (and, for hunter-gatherers, adaptive) tendency to attend disproportionately to threats and dangers.” He thinks that “the result is an information ecosystem systematically skewed towards catastrophe, conflict, and outrage.” The implication is that this negative skewing of reality is maladaptive in the modern world—something we should overcome with more enlightenment.

I disagree. If threatening, scary stuff is a kind of group mobilization fuel, then a good chunk of this catastrophism is politically rational. After all, it’s hard to mobilize a group by saying everything is peachy and getting better all the time. And once we realize that humans are not primarily dispassionate truth-seekers who care about accurately assessing intergenerational changes in health and income, but social primates who care about capturing each other’s attention, paying attention to what others are paying attention to, gaining and expressing sympathy for each other’s plights, signaling their competence and seriousness, being included in “important” conversations, demonizing their rivals, and saying things that are interesting and provocative, even our catastrophism has a certain kind of rationality to it. Positive trends are boring. Doomerism is exciting. Of course, all this doomerism takes a toll on our happiness. But as I’ve written about extensively (see here and here), we’re not pursuing happiness, so this shouldn’t count as evidence of human maladaptedness.

Mistakes Were Made (But Not By Me)

Dan lists a litany of bloody and catastrophic mistakes that have been made by humans throughout history, and I think this is where Dan is at his most persuasive. Humans have certainly done a lot of terrible shit. But while I acknowledge the tidal wave of stupidity that Dan is pointing to, it is important to remember that we are talking about the design of human nature, and how good we should expect that design to be—that is, whether the human mind is about as well-designed as the “hawk’s eye or the cheetah’s sprint,” as I put it in my post. When answering this question, we should not let ourselves get distracted by the specific failures of specific individuals, which are an inevitable part of life for any creature.

Predators often fail to catch their prey. Prey often fail to evade their predators. These failures cause death, which I’m told is a bad thing. But we shouldn’t conclude from these failures that predators and prey are dumb and irrational, or that they’re poorly designed for chasing and evading each other. “Haha, that gazelle just got eaten by a lion—what a dumbass!” In a world of fearsome competition and formidable constraints, deadly failures at the individual level and impeccable design at the species level are not mutually exclusive. Political revolutions often devour their children, but plenty of animals devour their children in the wild. The devouring does not necessarily make those animals, or their devoured children, maladapted.

Besides, even if we accept the Homo Stupidicus model that Dan is gesturing at, we have to ask ourselves the same question we asked previously: “Is there any reason to expect intellectuals to be less prone to these terrible blunders than the masses?” Given that many of the mistakes Dan cites were a result of intellectuals’ utopian visions, the answer is far from clear.

A Real Fixer Upper

Dan argues, contrary to my soul-crushing cynicism, that intellectuals often have a “genuine” motivation to fix the world. Rather than getting into a semantic debate about what it means to have a “genuine” motivation for something, I’d rather focus on what Dan and I seem to agree on: whenever people claim to be trying to fix the world, it is mostly because of deeper motives for esteem, prestige, admiration, etc. So if we want to understand this world-fixing business, we have to delve deeper into the prestige economy that gives rise to it. And once we delve deeper into that prestige economy, we will discover some serious grounds for pessimism. Because what gets a person prestige, and what fixes the world, are two very different things.

It is the appearance of world fixing to a prestige-granting audience—not objective world fixing in external reality—that intellectuals are striving for. And insofar as prestige-granting audiences do not actually know what fixes the world, or hold politically biased beliefs about what fixes the world, then intellectuals’ prestige striving will be uncorrelated with objective improvements in the world. You might even get a few cases where intellectuals get showered with virtue points for creating hell on earth. The disconnect between audience perceptions and objective reality is why I am more pessimistic than Dan about the world-fixing motivations of intellectuals. The lack of depth to these motivations is precisely what should make us skeptical that they will always lead to good outcomes, or that they are the main causes of moral and material progress throughout history.

Subscribe now

Enlightenment Now?

Aside from some differences in style, Dan and I probably agree on at least 90% of the substance, and I suspect he will be on board with most of what I’ve written here. That’s the beauty of good faith disagreement: it often reveals how little of it there is. And truth to be told, I’m just as enchanted by the ideals of the enlightenment as any other intellectual: it’s the animating force behind all my writing and researching and podcasting. So I really do get the emotional core of Dan’s essay. It’s what gets me out of bed in the morning.

But in spite of all the beauty and grandeur of the life of the mind, I cannot help but take a long hard look in the mirror and ask myself: “Is it all bullshit?” I think it’s important to ask ourselves this question, and if we’re going to ask it, we must be genuinely and uncomfortably open to the possibility that the answer is yes.

Does ‘wokeness’ exist at all? And if it did, why on earth would anyone get worked up over it? Over the past decade, many mainstream British liberals have reacted to the rise of ‘wokeness’, or what we prefer to call social justice ideology, by denying that anything of note is occurring. These kinds of denials have taken many forms. Among those we could affectionately term ‘centrist Dads’, there has been a rich seam of indignation that these sorts of questions are being raised at all. ‘Why do you care?’ ‘Why are you so obsessed with this?’ ‘Stop spending so much time on the internet!’ We have heard many responses along these lines.

On a more intellectual level, liberal objections have sought to characterize social justice ideology as just a particularly earnest and sincere form of liberalism. Surely, we hear, now is not the right time to look into this issue, given the very real threat of radical right populism? At worst, the desire to ask these questions at all is seen as alarming evidence of authoritarian tendencies. ‘It’s just idealistic kids’, we are told. Ignore it. It will all pass over in time. In academia, this has engendered a curious phenomenon: a near dearth of accounts which examine social justice ideology through an analytic lens.

In our new book, Beyond Woke and Anti-Woke: Explaining the Rise of Social Justice Ideology (Bristol University Press, 2026), we examine the seeming inability of liberals to describe their left flank in accurate terms (or even to admit that it exists at all). Over the past decade, liberalism has palpably lost ground to ‘woke’. Both are progressive ideologies but, in contrast to liberalism, social justice ideology emphasizes the overriding importance of identity and direct action. It extends the concept of harm far beyond previous limits, concerning itself particularly with the threats of emotional harm and harmful speech. These developments have brought social justice advocates into conflict with older liberal tenets: individualism, legalism, and freedom of speech and association.

This conspicuous gap in the scholarship is curious because, in academia, analytic approaches to ideology are common. To give a well-known example, there is an extensive literature on radical right populism which, over thousands of studies, examines the origins and trajectories of this ideology. Admittedly, many who are interested in the study of social justice ideology have not helped their cause. Since the mid-2010s, a self-described ‘heterodox’ movement has arisen that has sought to investigate the topic. This movement has produced some important works, yet has ultimately failed to grasp the social justice nettle. We have seen a great many polemical trade-press books and podcast episodes and many ‘free speech’ festivals at which speakers celebrate their ability to tolerate robust disagreement (but at which no one seems to disagree at all). Despite their initial energy and sense of purpose, many in this ‘heterodox’ space have ultimately abandoned all pretence of academic rigour. While surrendering to the temptations of audience capture might be good for Substack subscriptions, it makes it easier for liberals to dismiss these kinds of interventions. No smoke; no fire; nothing to see here.

Subscribe now

What, ultimately, explains the reluctance of liberals to acknowledge the existence of social justice ideology? We can think of some potential reasons: a certain hesitancy to expose the fractures in progressive movements, an unwillingness to be seen siding or affiliating with conservatives on certain issues, and (perhaps) fear of attack from radical activists. Sometimes, these kinds of motivations seem to be accompanied by the notion that progressive ideologies do not need to be explained. As classic theories of ideology contend, the ideologue regards their own worldview as an accurate depiction of reality and, therefore, without any need of further explanation. The overt similarities between liberalism and social justice ideology, and the obvious differences that separate them from conservative ideologies, encourage such thinking.

We find this state of affairs unfortunate, and it is what moved us to write our book. Just like radical right populism, we believe that social justice ideology should be studied with an analytic approach. Since the 2010s, social justice ideology has been the major newcomer in progressive ideological space and is notably different from liberalism. This development is fascinating and, rather than unevidenced polemics, the ideology deserves a serious programme of academic study, just like other ideologies.

In Beyond Woke and Anti-Woke, we explain the emergence of social justice ideology using statistical analysis of multiple surveys of UK and US public opinion, institutional theories of the political economy and morphological theories of ideology. Rather than having one cause, social justice ideology in fact reflects a wider demographic revolution. Mass higher education has transformed societies and, as women have entered public life, feminine-coded values of care and equality have become increasingly influential.

In particular, the crises of capitalism after the 2008 financial crash acted as catalysts for ideological change, giving social justice ideology mass appeal. Though our statistical analyses cast doubt on there being any direct relationship between adherence to social justice ideology and individual economic precarity (contrary to popular theories), we argue that economic crises helped discredit liberalism among younger groups. For corporations, the embrace of social justice ideology provided renewed legitimacy. By the 2020s, social justice ideology had become a major rival to liberalism and, notwithstanding the attacks of the second Trump administration, it remains a major force in progressive politics.

Of course, any such interpretations must be provisional. Beyond issues with replication, the lack of prior research on this topic makes conclusions unusually precarious. Will this change in the future? Though proponents of a spatial hypothesis expect that academic supply will inevitably meet any gaps that appear in the intellectual market, we are less optimistic. Academic fields are path dependent and, therefore, tend to follow their own logic. If they have been closed off to certain lines of inquiry in the past, there is a certain inevitability that they will continue to be so. Moreover, we know from survey data that progressives are numerically predominant in universities. Inevitably, this will create pressures from within fields and disciplines to maintain existing path dependencies.

Perhaps there is need for a cultural change in this area. Contrary to the fears of some, analysing progressive ideologies does not imply that one regards them as a problem to be solved. Instead, this sort of investigation should be entirely conventional in academia; one identifies a gap in the research and, using established theories and methods, arrives at findings. In the case of social justice ideology, its crucial influence on institutions and policymaking adds to the justification for such a research agenda.

This, we argue, would not only lead to a healthier academia; it would lead to a healthier liberalism.

In a characteristically insightful and entertaining essay, David Pinsof argues that intellectuals greatly overestimate how many of the world’s problems stem from popular misunderstandings. In reality, Pinsof argues, people are highly rational and well-informed about their interests. This is what we should expect on evolutionary grounds. “Show me an animal that has succeeded in surviving and reproducing in a hostile environment for millions of years, and I will show you a rational animal.” It is also supported by extensive evidence about the rationality and accuracy of human cognition.

In Pinsof’s worldview, even the dreaded cognitive “biases” that psychologists love to tell us about function as adaptive mechanisms that help us survive and thrive. Confirmation bias, for example, provides us with intellectual ammunition for persuasion and reputation management, while overconfidence and self-serving illusions help us win friends and influence people.

Why, then, do intellectuals so often chalk up the world’s problems to mass ignorance and irrationality? Partly, the narrative is simply self-serving. It is intellectuals, after all, who promise to liberate us from misunderstanding. They are our professional understanders.

But it’s also because they confuse our expressed motives with our real goals. Sure, Pinsof concedes, we look pretty stupid and misinformed relative to the high ideals and noble ambitions that we say we have. If we’re chasing objective truth, impartial justice, and effective altruism, we’re not doing a good job. But those goals are just elaborate fictions, self-serving public relations cooked up to make us look good. Our real goals, our hidden motives, are very different. We’re chasing the kinds of grubby rewards you would expect of apes forged in Darwinian competition: status, reputation, power, sex, and resources. And relative to those ambitions, we’re smart and sophisticated.

This analysis reframes many apparent examples of stupidity as strategies. For example, “tribalism” isn’t a cognitive error to be remedied by debiasing and education; it’s a winning strategy among groupish primates who care more about power and prestige than truth or justice. Ineffective altruism and slacktivism don’t result from miscalculating the most effective ways to help others; they help status-seeking activists buy noble reputations at a discount.

Unsurprisingly, this perspective leads Pinsof to a bleak conclusion. If most of the world’s problems result not from misunderstandings but from conflicting incentives, intellectual enlightenment cannot save us. And even if it could, nobody really cares about solving the world’s problems anyway:

Not every problem has a solution. Some things cannot be fixed. And once you come to the bracing realization that we have no deep desire to fix our broken world, you’ll realize that our problem is that we have no problem. What’s broken is that nothing is broken. The study of human nature is, all too often, the study of the hole we’re stuck in… In the end, the only misunderstanding is that there’s been a misunderstanding.

A Darwinian Defence of the Enlightenment

It’s a beautifully cynical, Pinsofian analysis—and one that, I think, gets a lot right.

Nevertheless, it is too optimistic about our baseline rationality. Yes, we are savvy and strategic primates pursuing goals we’d rather not admit, even to ourselves. But we’re also riddled with costly cognitive biases, maladapted to the modern world, and in need of enlightenment by intellectual knowledge that we often find deeply counterintuitive.

It is also too pessimistic. Some people really are motivated to fix our broken world, and in some cases, they make genuine progress. The motivation is never very deep or pure—no straight thing was ever made from the crooked timber of humanity—but it’s not merely a deceptive story, either.

There are many holes we will never escape from. There is an unavoidably tragic aspect to the human condition. But when scaffolded by the right incentives and error-correction mechanisms, we can draw on intellectual knowledge and cooperation to climb out of the worst pits we find ourselves in.

You can’t understand much of humanity’s significant progress over the past several centuries—in life expectancy, living standards, wealth, health, infant mortality, freedom, political governance, and so on—without embracing this fundamental optimism of the Enlightenment. Or so I will argue.

Evolutionary Expectations

Before getting into the details, it’s worth stepping back and scrutinising Pinsof’s assumptions about evolution and human rationality. He says,

“The default assumption of every intellectual should be that the human mind is about as well-designed as the hawk’s eye, the bat’s sonar, or the cheetah’s sprint.”

Our species complicates this default assumption in two ways.

A Uniquely Unique Animal

First, although all species are unique, it’s not just human chauvinism to think that we’re uniquely unique, a genuinely new kind of animal.

There is no single quality responsible for this—no magic bullet that set our ancestors on a novel evolutionary pathway. Instead, there is a set of interacting traits connected to our unique capacities for cognition (how we think and reason), cooperation (how we work together), and culture (how we share and accumulate information). Through such abilities, we have acquired unprecedented powers to design and redesign our environments, but we have also become vulnerable to novel risks and failure modes.

To take only one example, no other species is anywhere near as dependent on lifetime learning as we are, including extensive “social learning.” To achieve our goals, we rely on information acquired from others (parents, family, friends, allies, teachers, shamans, priests, Substackers, etc.), typically because they intentionally share it with us through language and other forms of communication.

Given this reliance, evolution has endowed us with highly sophisticated social learning mechanisms. In this sense, Pinsof is right that evolutionary theory correctly predicts rationality and adaptation. We’re skilled at extracting knowledge from others while minimising the risks of misinformation and deception. Even from a young age, we instinctively evaluate the plausibility of what we’re told, assess people’s reliability and honesty across different domains, and insist on persuasive arguments for surprising claims.

Nevertheless, such extensive social learning also creates novel vulnerabilities that won’t be illuminated by analogies to the hawk’s eye, bat’s sonar, or cheetah’s sprint.

Most obviously, it means that reflection on human evolution should never be used to discount the importance of ideas. We evolved to be a species dependent on good ideas—on the knowledge, wisdom, and understanding that we acquire from others. If such ideas are misleading or deceptive in ways we can’t anticipate or detect, even optimal learning mechanisms won’t prevent us from being misinformed in costly and sometimes catastrophic ways.

Before the Neolithic Revolution, this vulnerability wasn’t very pressing for most humans. The challenges hunter-gatherers faced were mostly local and small-scale: which plants are edible, which animals migrate, which group members are trustworthy, and so on. This meant they could often cross-check what they were told against direct experience.

Moreover, because our core intuitions evolved over hundreds of millennia in response to hunter-gatherer lifestyles, people’s instinctive bullshit detectors were broadly reliable in such domains. Whenever they encountered claims that seemed implausible or outlandish—that is, counterintuitive—they could usually safely dismiss them, or at least insist on practical demonstrations of their veracity.

Finally, because their social networks were mostly face-to-face, highly interdependent, and largely egalitarian, high-stakes deception was typically risky and counterproductive. When everyone knows everyone extremely well, and power is broadly distributed, it’s easier to discover and punish big lies. And when everyone depends on everyone else for the most basic necessities of survival, the social costs of getting caught lying can be astronomical.

Of course, hunter-gatherers believed plenty of preposterous falsehoods about matters beyond their experience—for example, about the broader cosmos, their ancient history, or the character of rival tribes. But such myths were generally costless and adaptive. When you lack the ability to influence the world beyond your immediate, day-to-day existence, you can believe whatever you want about it, which is exactly what they did.

The New World

Well, things have changed. The second reason humans complicate the link between evolution and rationality is that the modern world we must navigate is unimaginably more vast, complex, and unequal than hunter-gatherer environments. As John Dewey observed a century ago,

“The local face-to-face community has been invaded by forces so vast, so remote in initiation, so far-reaching in scope and so complexly indirect in operation that they are, from the standpoint of the members of local social units, unknown. . . . They act at a great distance in ways invisible to [them].”

Natural selection adapts organisms to their environments. When these environments change, such adaptations can become “mismatched.” This is why things aren’t going so well for polar bears.

In the human case, evolutionary mismatch is often invoked to explain relatively mundane things, such as why so many of us are obese. As the familiar story goes, sugar and fat were scarce in ancestral environments, so we evolved to crave them. In modern capitalist societies, they are abundant. So we gorge on cheesecake and pizza served to us by profit-seeking companies that place no value on our welfare. There is no savvy strategy behind such overeating. Most of us are simply heavier and unhealthier than we would like to be.

This basic lesson generalises to many other contexts, including those where our maladaptation is harder to observe than our fatness. The most important of these is modern politics.

Political Mismatch

The scale and complexity of the modern environment that bears on political debate are mind-boggling. Hundreds of millions of strangers are enmeshed in interacting economic, political, and institutional forces that bear no resemblance to the small-scale worlds we evolved in.

Although it’s important not to overstate the problem of mismatch here—popular talk of static “stone-age minds” obscures how we evolved to be highly adaptable and flexible—it’s equally important not to ignore the severity of the challenges.

First, the modern world radicalises our reliance on social learning. When forming beliefs about topics relevant to modern politics, we almost always lack the ability to cross-check what we’re told against our experience, either because it is too distant in space and time or because the topics concern abstract phenomena (GDP, inflation, demographic trends, economic growth, etc.) that no one can directly experience.

Second, the intuitions most people bring to understanding modern societies are systematically misleading.

We evolved to be highly skilled at forming alliances, reading intentions, tracking reputations, and playing local status games. In contrast, neither our evolutionary endowment nor first-hand experiences prepare us to understand large-scale systems characterised by emergent properties, distributed processes, and incentives. So we anthropomorphise institutions and frequently default to moralised, intention-based narratives that posit villains rather than incentives and structural constraints.

When pharmaceutical prices rise, we assume that greedy executives are to blame rather than laws, regulations, and insurance markets. When housing or renting becomes unaffordable, we blame the avarice of developers and landlords rather than building restrictions, permitting processes, and construction costs. Such tendencies served our ancestors well. In hunter-gatherer societies, it’s reasonable to trace significant events to identifiable agents with familiar goals, and to link good and bad social outcomes to good and bad intentions. Invisible-hand coordination and emergent order are, therefore, deeply counterintuitive.

Similarly, zero-sum thinking makes sense for hunter-gatherers. When you live at the subsistence level, one person’s dramatic gains likely mean someone else’s dramatic loss. Consequently, we struggle to comprehend how modern trade and innovation could make everyone better off, especially when gains are unevenly distributed or delayed. In fact, the very idea that something called “wealth” can be created is a profound theoretical discovery that conflicts with common sense. The more natural view, which modern economics education tries to shake people out of, is that there is a fixed set of goods to be distributed.

Third, the modern information environment through which people attempt to learn about this strange, new world and overcome their default ignorance and confusion is more of a hindrance than a help.

Most obviously, it is shaped by extremely well-funded propaganda campaigns by powerful strangers who profit from other people’s ignorance. Even if such propaganda is unsuccessful, as it often is, its presence can breed pervasive mistrust that prevents the uptake of trustworthy information, causing people to place greater weight on their personal—and highly unreliable—intuitions.

Even more importantly, the modern media environment in free societies is organised around intense competition for audience attention and engagement. When combined with our deep-rooted negativity bias—our evolved (and, for hunter-gatherers, adaptive) tendency to attend disproportionately to threats and dangers—the result is an information ecosystem systematically skewed towards catastrophe, conflict, and outrage.

The predictable consequence of this is that people develop mental pictures of reality far more negative than the objective facts warrant. They overestimate poverty, crime rates, and many other social pathologies and dangers, and believe most trends are going in the wrong direction. In affluent liberal democracies, people are not just largely oblivious to progress. Their minds invert reality, often treating the most peaceful and prosperous societies in human history as dystopian hellscapes.

The result of all this is pervasive ignorance and misperception. The facts and complexities of the modern world are substituted in people’s heads with cartoonish, catastrophising myths.

The Rational Irrationality Objection

If this analysis is correct, it suggests that mass ignorance and misperceptions are not figments of intellectuals’ self-serving imaginations. Evolution made us rational and well-adapted—but to a world that no longer exists. In the modern world, confusion and misunderstanding are the default.

Nevertheless, there is a popular line of reasoning that concedes the existence of mass ignorance but insists that it is “rational.” To introduce a bit of jargon, it acknowledges that most people are not “epistemically rational”—they are doing a terrible job forming accurate beliefs about reality—but it argues that such epistemic failures are “instrumentally rational”. In line with Pinsof’s perspective, it treats widespread error and delusion as an adaptive response to people’s practical circumstances.

One influential theory of this kind comes from the work of economists like Anthony Downs and Bryan Caplan. It points out that in large-scale modern democracies, an individual’s vote makes practically no difference to electoral outcomes. This means people have no incentive to become well informed. They have no skin in the game. On the other hand, endorsing political beliefs that are emotionally gratifying or that signal one’s tribal loyalties can be highly beneficial. So rational individuals opt for ignorance and (epistemic) irrationality.

This analysis could be strengthened by Pinsof’s “Alliance Theory” of political belief systems, which posits that people’s participation in politics is not rooted in a desire to form accurate beliefs. Instead, we’re tribal propagandists. Our beliefs are downstream of the alliances and rivalries we form, and the biased, hypocritical arguments we construct to make our allies look good and our rivals look bad.

Both perspectives are insightful, but they also go too far.

Sometimes Ignorance and Irrationality Are Just Ignorance and Irrationality

One problem for the “rational ignorance” perspective is the prediction that ignorance and misperceptions will evaporate when people have skin in the game. This is wrong.

The history of modernity is littered with examples of people making catastrophic decisions based on deranged, inaccurate worldviews in high-stakes contexts. The Nazis really believed in an elaborate Jewish conspiracy, which led them to undertake self-defeating decisions, such as diverting crucial wartime resources to mass genocide. As I will return to below, communists throughout the twentieth century genuinely believed in various myths about human nature and economics, which led to repeated catastrophes, many of which engulfed the revolutionaries who brought such regimes into existence.

For less severe examples, one need only look at the policy track records of populist politicians in modern democracies to see that people often make bad decisions based on ignorance and misperceptions, even when they have strong incentives to perceive reality accurately.

The idea that political cognition improves dramatically as stakes increase is not well supported by the historical record. And once you reflect on the vastness, complexity, and inaccessibility of the modern world, this shouldn’t be very surprising. When discovering the truth is extremely challenging, merely increasing people’s incentive to discover it won’t secure success.

Another problem with the “rational ignorance” perspective is the assumption that people know their individual vote has no impact on political outcomes and so “decide” to be ignorant and misinformed. As Jeffrey Friedman points out, this isn’t well-supported by evidence. Instead, people appear to be radically ignorant, not rationally ignorant. Because they don’t instinctively appreciate the sheer scale of the modern world, they dramatically overestimate the impact of their vote, and they treat political knowledge as much more accessible than it really is.

This analysis helps to explain many features of political psychology that sit uneasily with the “rational ignorance” perspective. The intensely negative, catastrophising worldviews that many people develop often just make them sad, distressed, and demotivated. They experience politics as aversive and anxiogenic. They sometimes damage close relationships with friends and family members. Much of this looks more like sincere participation than tribal signalling optimised for maximising emotional or social rewards.

None of this is to deny that people approach politics with a “tribal” mindset. There is considerable insight in Pinsof’s analysis that politics is rooted in alliances, rivalries, and self-serving (well, alliance-serving) “propaganda,” as well as in the popular idea that much political participation is performative, concerned more with tribal signalling than sober policy analysis.

However, the problem with such proposals is that the leaders and tribes we support and oppose are not independent of—in technical jargon, they’re not “exogenous” to—our political beliefs, so they cannot fully explain such beliefs. We choose leaders, allies, rivals, and enemies based on the pictures in our heads. If those pictures are systematically warped by misleading intuitions, mistrust, and negativity bias, the same will apply to our judgements about which leaders and allies promote our interests.

Put simply, someone with an accurate, evidence-based worldview will support very different political leaders and tribes than someone whose worldview is constructed from “common sense” intuitions interacting with their TikTok feed.

In general, political ignorance and misperceptions aren’t always or even commonly the product of savvy, evidence-based cost-benefit analysis or 4D Darwinian chess. They’re often downstream of the profound challenges of acquiring counterintuitive knowledge in a hostile and misleading information environment.

Subscribe now

The Role of Intellectuals

This suggests a more optimistic assessment of the value of “intellectuals” in the broad sense of that term (scientists, statisticians, academics, etc.), and of the kinds of knowledge they can provide, ranging from carefully collected data to rigorous scientific inquiry. To successfully navigate the modern world, we need to be enlightened by such knowledge. It won’t fall into our lap if we let our evolved psychologies run on autopilot. Our default condition is one of epistemic darkness.

This optimism is, or at least should be, uncontroversial when it comes to the knowledge associated with the natural and medical sciences. Hundreds of millions of people died throughout history from diseases we have now eradicated thanks to discoveries about vaccines and other miracles of modern medicine and public health. We couldn’t rely on Darwinian adaptations to secure such knowledge. We needed rigorous, institutionally supported inquiry through which we could learn truths that are often highly counterintuitive.

The real controversy concerns whether intellectual knowledge can correct costly ignorance in domains like politics and collective organisation.

Here, scepticism is understandable. It’s much more challenging to conduct rigorous science in these domains, and prominent ideas and theories often function more like intellectual fashions governed by the subjective, internal criteria upheld by the intelligentsia than like scientific hypotheses evaluated by objective measures of predictive success.

For this reason, the practical track record of these ideas has often been negative, and in some cases disastrous. Despite concerted and ongoing obfuscation of this fact by many left-wing intellectuals, the clearest example lies with Marx, who, alongside many later generations of communist intellectuals and activists inspired by his work, argued that self-interest and social competition were not essential features of human nature but contingent products of exploitative economic systems like capitalism, feudalism, and slavery. This and countless other foolish ideas, such as the notion that law and conventional morality under capitalism are mere “bourgeois prejudices,” played a major and undeniable role in many of the worst human catastrophes of the twentieth century.

These catastrophes can’t be understood as moral abominations that nevertheless advanced the strategic interests of those who spread them. Most of the true believers who fought for communist revolutions in countries like Russia, China, Korea, and Cambodia were quickly victimised by the systems they helped create. They weren’t just playing cynical adaptive games. They were catastrophically misinformed about reality in ways that got themselves and countless others killed.

Notice, however, that one should not conclude from such disasters that intellectual ideas don’t matter. They matter enormously. But wouldn’t it be strange if they could only have negative consequences?

The Achievements of Liberalism

In fact, one can find many examples throughout history of intellectual achievements concerning society and politics that have had extremely beneficial consequences.

For example, as Steven Pinker, Joseph Heath, Jonathan Rauch, and many others have documented, one cannot understand the emergence of modern liberalism and the unique social and political successes of liberal states without appreciating how complex, counterintuitive intellectual discoveries informed institution-building. From at least Hobbes onwards, a tradition of intellectual inquiry—including Locke, Hume, Montesquieu, Smith, Kant, and many other Enlightenment thinkers—drew attention to two major theoretical insights.

The first was that human societies are pervaded by what social scientists now call “collective action problems”: situations where individuals acting on their rational self-interest are led to engage in collectively self-defeating behaviour that leaves everyone worse off.

For example, Hobbes observed how, in the absence of enforceable laws and contracts, people who would benefit from mutual cooperation would be driven towards pre-emptive aggression, fearing exploitation or cheating by others. Insights with a similar structure were later used to explain the value of political regimes that uphold religious and political tolerance, enforce extensive systems of individual rights, protect free speech even for dangerous and heretical ideas, maintain open trade between nations, and more.

The second insight was that institutions can be constructed to channel self-interest and social competition away from predation and violence towards beneficial outcomes.

For example, Smith demonstrated how regulated market competition could transform the self-interest of bakers and brewers into the efficient production of goods for others. In the political domain, Montesquieu and Madison explored how political systems could be organised to make ambition counteract ambition. And in the domain of knowledge, many scientists and philosophers explored how formal societies and norms could be crafted to counteract individual biases and allocate “credit” only to those who made genuine discoveries. The core discovery across these diverse contexts was that specific systems of norms and institutions can convert human self-interest and ambition into innovation, investment, knowledge, and political accountability.

In both cases, these insights were genuine theoretical discoveries that sharply contradicted most people’s intuitions. Trusting in self-interest, social competition, and decentralised markets to coordinate economic activity; relinquishing power to people with radically different political or religious views; tolerating dangerous and offensive speech—these ideas don’t come naturally to human beings. They are insights that must be achieved.

Of course, the insights alone don’t change anything. Merely recognising the existence of a collective action problem doesn’t free one from it. And merely understanding that institutions can channel ambition into cooperation doesn’t create them. Nevertheless, precisely because these insights take humans as they are, not as we’d like them to be, and point to possibilities that leave everyone better off, not just some people, they can guide institutional design and tinkering, helping humanity gradually escape from the poverty, ignorance, and conflict that are our default state.

Notice, however, that for this progress to be possible, societies require a critical mass of people who appreciate these insights. Similarly, there must be enough people who have a grasp of the basic facts and trends demonstrating that such insights work—that, as a consequence of liberal institutions guided by intellectual insights, much of humanity has experienced objective progress along countless dimensions, including wealth, health, freedom, opportunity, governance, and much more.

The Desire to Fix the World

Reflecting on this history puts pressure on Pinsof’s pessimistic judgement that nobody really wants to fix our broken world.

Once again, there is more than one grain of truth here. Humans are unavoidably self-interested and competitive, and altruistic motivations are inevitably limited and accompanied by a large dose of selectivity and hypocrisy. This is what we should expect on evolutionary grounds, and it is confirmed by extensive historical evidence, not least the many examples where revolutionaries championing justice have quickly turned into despots after taking power.

Nevertheless, the historical record suggests that the deep human craving for esteem and honour can also be channelled into genuinely noble pursuits. As we created liberal societies that increased the scale of cooperation and the costs of predation, we also created conditions that made the pursuit of prestige—of admiration and deference—more profitable. As Will Storr documents in The Status Game, this channelled insatiable human ambition and social competition towards impressing others through demonstrations of competence and virtue, fuelling modern science, innovation, and social justice.

We reward those who try to fix the world, produce novel insights, and advance other people’s interests. At the same time, we are sensitive to the possibility that such motivations aren’t genuine—that people care only about the personal rewards, not the high ideals. Nichola Raihani calls this the “reputation tightrope”: to earn a noble reputation for performing good deeds, those deeds must flow from the right motives, not reputational ones.

As a consequence, in societies that consistently reward prosocial behaviour, people tend to internalise their motivations to help others. The best way to convince others that you want to help them and fix our broken world is to develop a genuine passion for doing so.

Such passions are never pure or extremely deep. They must be scaffolded by the right incentives, and they can disappear if incentives suddenly change—hence the many justice-championing revolutionaries who lose their love of humanity when they acquire power. But it is genuine and sincere nonetheless, and you can’t understand humanity’s progress over recent centuries without appreciating its reality—from the scientists and doctors who devoted their lives to understanding and combating disease, to the reformers who fought for social justice against slavery and oppression, to the entrepreneurs who created technologies that lifted countless people out of poverty.

Conclusion

The truth is messy and complex. We are rational creatures whose apparent “stupidity” is often a symptom of hidden strategies, but we are also maladapted to modernity’s vastness and complexity. In this strange new world, ignorance is our default, our intuitions mislead us, and the information environment exacerbates our confusion. To escape this bleak situation, we must unlearn our “common sense”. We need to be enlightened by insights and knowledge that only systematic, intellectual inquiry can provide.

Such inquiry has demonstrably improved the human condition. Liberal norms and institutions, products of hard-won, counterintuitive discoveries, function to channel our self-interest and ambition into cooperation and progress, helped along by a craving for prestige that can be—and has been—directed towards noble pursuits that have made the world measurably better.

And yet, at present, a shocking number of people are ignorant of this progress, and of the insights that underpinned it. If you read survey data or listen to the speeches of some of the West’s most popular politicians, you discover that many people sincerely believe that things have been getting worse.

This is a big misunderstanding.

To correct it, we must insist on the value of intellectual insights and carefully collected data. We must acknowledge that too many people are ignorant and confused about the world they inhabit, and celebrate those who aim to change that.

Further Reading:

• David Pinsof is one of my favourite writers and social scientists. I’m not sure how much he would ultimately disagree with my arguments here.

• On the role of counterintuitive liberal insights in the Enlightenment, I’ve been highly influenced by Joseph Heath, Jonathan Rauch, and Steven Pinker.

• On how status competition can be (and has been) channelled into cooperation and progress, see Will Storr and Nichola Raihani.

• On social complexity, political modernity, and evolutionary mismatch, see Walter Lippmann and Pascal Boyer.

• On why people are sincerely misled about the fact of modern progress, see Max Roser and Hannah Ritchie.

• How prevalent is misinformation?

• What even is “misinformation”?

• Is there a difference between politics and science?

• How impactful are propaganda, influence campaigns, and advertising?

• What impact has social media had on modern democracies?

• How worried should we be about the impact of generative AI, including deepfakes, on the information environment?

• The “liar’s dividend”

• Whether ChatGPT is more accurate and less biased than the average politician, pundit, and voter.

Links

• Sacha Altay

• “Misinformation Reloaded? Fears about the Impact of Generative AI on Misinformation are Overblown” Felix M. Simon, Sacha Altay, & Hugo Mercier

• “Don’t Panic (Yet): Assessing the Evidence and Discourse Around Generative AI and Elections” Felix M. Simon & Sacha Altay

• “The Media Very Rarely Lies” Scott Alexander

• “How Dangerous is Misinformation?” Dan Williams

• “Scapegoating the Algorithm” Dan Williams

• “Is Social Media Destroying Democracy—Or Giving It To Us Good And Hard?” Dan Williams

• “Not Born Yesterday: The Science of Who We Trust and What We Believe” Hugo Mercier

• Joseph Uscinski

• “Durably Reducing Conspiracy Beliefs Through Dialogues with AI” Thomas H. Costello, Gordon Pennycook, & David G. Rand

• “The Levers of Political Persuasion with Conversational AI” Kobi Hackenburg, Ben M. Tappin, et al.

• Ben Tappin

Chapters

• 00:00 Understanding Misinformation: Definitions and Prevalence

• 04:22 The Complexity of Media Bias and Misinformation

• 14:40 Human Gullibility: Misconceptions and Realities

• 27:28 Selective Exposure and Demand for Misinformation

• 29:49 Political Advertising: Efficacy and Misconceptions

• 35:13 Social Media’s Role in Political Discourse

• 40:50 Evaluating the Impact of Social Media on Society

• 42:44 The Impact of Political Content on Social Media

• 46:57 The Changing Landscape of Political Voices

• 51:41 Generative AI and Its Implications for Misinformation

• 01:03:46 The Liar’s Dividend and Trust in Media

• 01:14:11 Personalization and the Role of Generative AI

Transcript

• Please note that this transcript was edited by AI and may contain mistakes.

Dan Williams: Okay, welcome back. I’m Dan Williams. I’m back with Henry Shevlin. And today we’re going to be talking about one of the most controversial, consequential topics in popular discourse, in academic research, and in politics, which is misinformation. So we’re going to be talking about how widespread is misinformation? Are we living through, as some people claim, a misinformation age, a post-truth era, an epistemic crisis?

How impactful is misinformation and more broadly domestic and foreign influence campaigns? What’s the role of social media platforms like TikTok, YouTube, like Facebook, like X when it comes to the information environment? Is social media a kind of technological wrecking ball which has smashed into democratic societies and created all sorts of havoc? And also what’s the impact of generative AI when it comes to the information environment?

Both when it comes to systems like ChatGPT, but also when it comes to deepfakes, use of generative AI to create hyper-realistic audio, video, and images. Fortunately, we’re joined by Sacha Altay, brilliant heterodox researcher in the misinformation space, who pushes back against what he perceives to be simplistic and alarmist takes concerning misinformation.

So we’re going to be picking Sacha’s brain and just more generally having a chat about misinformation, social media, and the information environment. So Sacha, maybe just to kick things off, in your estimation, if we’re keeping our focus on Western democracies, how prevalent is misinformation?

Sacha Altay: Hi guys, my pleasure to be here. So it’s a very difficult question because we need to define what is misinformation. So we’ll first stick to the empirical literature on misinformation and look at the scientific estimates of misinformation. For that, there are basically two ways or three ways to define misinformation. One of them is to look at fact-checked false news.

So false news that have been fact-checked by fact-checkers as being false or misleading. And by this account, misinformation is quite small on social media, like Facebook or Twitter. It’s in between 1 and 5% of all the content or all the news that people come across. So according to this definition, it’s quite small. There is some variability across country. For instance, it seems to be higher in country like, I don’t know, the US or France than the UK or Germany.

There is another definition which is a bit more expansive because the problem with fact-checked false news is that you rest entirely on the work of fact-checkers and of course fact-checkers cannot fact-check everything and not all misinformation is news. So you see the problems. So another way is to just look at the sources of information and you classify them based on how good they are and how basically how much they share reliable information, how much they have good journalistic practice, et cetera. And the advantage of this technique is that you can have a much broader range because you can have, I don’t know, 3,000 sources of information that share information. And usually it broadly like most of the information that people see. And according to the definitions, misinformation is also quite small. So the definition is just for misleading information that comes from the sources that are judged as unreliable. And by this definition, misinformation is also quite small. Again, it’s like about like one to 5% of all the news that people encounter.

But then of course, the problem is not all the information that people encounter comes in this form. And for instance, some of it can come in terms of like images or all the sorts of things. And so this broadens the definition of misinformation. So some people think that when you broaden this definition, you have much more misinformation. My reading is that when you broaden this definition, you actually include so much more information that you increase the denominator. So of course, there’s going to be more misinformation, but because the denominator is larger, the proportion is going to be pretty much the same. But that’s an empirical question. So let’s say to sum up that it’s smaller than people think, according to the scientific estimates.

Henry Shevlin: If I can just come in here, a point that Dan you’ve emphasized in our conversations to me, and I think Scott Alexander has also emphasized in a great blog post called The Media Very Rarely Lies, is that a lot of what people think of as misinformation is just true information selectively expressed or couched in a way that naturally leads people to maybe form false beliefs but doesn’t involve presentation of falsehoods. Does that sort of feature in any of these sort of more expansive definitions of misinformation? Is it possible to create definitions that can capture this kind of deceptive, intentionally deceptive but not strictly false content?

Sacha Altay: I’d say that when you look at the definitions based on the sources, if a source is systematically biased and systematically misrepresent evidence and stuff, they are going to be classified as misinformation. I think the problem and the more subtle point is that these sources are not very important because people don’t trust them very much. But the bigger problem is when much more trusted sources who have a much larger reach, like I don’t know the BBC or the New York Times, they are accurate like most of the time, but sometimes and on systematic issues like I don’t know, they can be wrong. And that’s the bigger issue because they are right most of the time. So they have a big reach, they have big trust, but they are wrong sometimes. And that’s the problem.

Dan Williams: But I think just to focus on that observation of Henry’s, you might say, well, they’re accurate most of the time, but nevertheless, you can have a media outlet which is strictly speaking accurate most of the time with every single news story that it reports on. But because of the ways in which it selects, omits, frames, packages, contextualizes information, nevertheless end up misinforming audiences, even if every single story that they’re reporting on is on its merits, sort of factual and evidence-based.

I mean, I think the way that I understand what’s happening in this broader debate about the prevalence of misinformation is round about 2016 when we had Brexit in the United Kingdom and then the first election of Donald Trump, there was this massive panic about misinformation because many people thought maybe that’s what’s driving a lot of this support for what gets called like right-wing authoritarian populist politics. And around that time when people were thinking of the term misinformation, they were kind of thinking of fake news in the sort of literal sense of that term. So false outright fabricated information presented in the format of news. And as you pointed out, when researchers then looked at the prevalence of that kind of content, which you don’t really find when it comes to establishment news media for the most part, like there are always gonna be exceptions, that stuff is pretty rare.

And then one of the responses to that is to say, okay, if you’re only looking at like outright fake news, then you’re missing all of these other ways in which communication can be misleading by being selective, by omitting relevant context through framing, through kind of subtle ideological biases.

And then my view on that is, well, once you’ve expanded the term to that extent, and you’ve got this really kind of elastic, amorphous definition, it becomes really kind of analytically useless. Like you’re just bundling together so many different things. And that kind of content is also really pervasive in my view, within many of our establishment institutions, including within the social sciences. But Sacha, it sounds like you don’t necessarily want to endorse that last point. You seem to be thinking, even if you do have this kind of very broad definition of misinformation, we can still say that it’s a pretty fringe or pretty rare feature of the information environment. Is that fair? Am I understanding you right? Or is there something different going on?

Sacha Altay: I think I would agree with you that if the simple fact of framing information or having an opinion, like any scientist, even in the hard sciences, they have some theories that they prefer, they are more familiar with certain frameworks, and so they are going to be biased anyways. Scientists are humans, they are biased, but calling physics or the theory of relativity or whatever misinformation because it omits certain facts that it cannot accommodate or whatever, I think it’s far-fetched. I think it goes too far. So yeah, I would agree that if you use this broad definition of misinformation, then it’s very widespread. But then, yeah, even theories in physics would be misinformation because they cannot be completely objective.

I think science works not because scientific individuals are perfect, etc., or even because one theory is perfect, but because as a whole and as an exercise of arguing, etc., we get better and a little bit closer to the truth. But still, we are not getting at the truth and we cannot avoid the mistakes that you’re pointing.

Henry Shevlin: If I just want to push back a tiny bit, it seems to me, so obviously there’s this point here that, you know, all theory is value laden, the kind of physics point that I think is maybe true, but not very interesting. But I think there is maybe something in the middle here that is what I worry about, which is cases where there might be really quite, quite deliberate pushing of an agenda, a realization by a media provider that they are generating maybe inaccurate views, but they’re doing so just through reporting factual things.

So one example, Dan, that you’ve given before is that most of the kind of what we think of as misleading anti-vax discussion just reports on true factually accurate but rare vaccine deaths, but just reports on them in a very regular fashion. In the same way, you might think that selective reporting of certain kinds of violent incidents, whether it’s terrorism, police shootings, leads systematically to overestimation of the incidence of this kind of phenomena by the public or increased worries about its prevalence in a way that I think is perhaps worrying and politically objectionable, right? I think we might say, hang on, it is bad that we give so much press coverage to event type X rather than event type Y. And we know that this leads the public to overestimate the prevalence of event type X compared to event type Y. So I think there’s something in between the sort of, well, even physics is biased and the view of misinformation as, you know, strictly speaking lies. This kind of third category. I don’t know if that, I defer to you both as misinformation experts, but it seems that that is a worrying category.

Sacha Altay: I think you’re totally correct. And that’s what the field of misinformation has been proposing, like just for instance, classifying headlines based not on whether they are true or false, but whether they will create misperceptions after you have read them. And so researchers are saying, for instance, that we should classify as misinformation headlines such as, “a doctor died a week after getting vaccinated and we are investigating the cause.” And I think I disagree with this. I disagree with this thing that we should classify this.

What you were suggesting, Henry, was a bit different, is that it needs to be systematic. If you systematically misrepresent vaccine side effects, then it becomes problematic. But reporting on vaccine side effects and their possible negative effects is normal. And I think it’s healthy that news outlets are able to talk about and cover negative effects of vaccines, even if after reading the headlines, you have more negative opinions about vaccines, which is not supported by science, et cetera—they should be able to do that and they should do that. But if it’s systematic, as you say, I think it becomes more problematic. But I do think that when the bias is very strong in some of the definitions of misinformation based on the source, they would be classified as misinformation sources like Breitbart, et cetera. They are systematically extremely biased towards, I don’t know, these kind of things. And so they would be classified as misinformation.

Dan Williams: I think sort of one of the worries that I have though is who decides what constitutes systematic bias and bias about what? I think there’s a real kind of epistemological naivety that I often encounter with misinformation researchers where it’s like, you’re reporting accurate but unrepresentative events when it comes to vaccines. So we can call that misinformation. And then it’s like, well, as Henry mentioned, well, what about police killings of unarmed black citizens in the US. There’s a vast amount of media coverage of those sorts of events. Someone might argue that they are, statistically speaking, rare and unrepresentative, and that large segments of the public dramatically overestimate how pervasive those sorts of occurrences are.

And I think you go through many, many examples like that. And for me, the lesson to draw from that is not that, therefore, there are no differences in quality when it comes to the different media outlets in the information environment, like of course there are, but I also think like there’s such a thing as politics and there’s such a thing as science, where you’ve got scientists who attempt to acquire a kind of objective intellectual authority on certain things, and we should be very careful not to kind of blur the distinction between those two things.

I think when we’re talking about media bias in this really expansive way, where we’re not saying, okay, you’re just making shit up, but we’re saying you’re being selective in terms of which aspects of reality you’re choosing. For me, that’s a really important debate, but it’s a debate that happens within the context of politics and democratic debate and deliberation and argument. And I think sometimes I encounter misinformation researchers who treat that as if it’s just, it’s a simple sort of technocratic scientific question. Like we can quantify the degree to which the New York Times is biased or we can objectively evaluate the degree to which different kinds of outlets approximate the objective truth when it comes to their systematic coverage. And I get a little bit kind of squirmy when we get to that point, because I think that there’s just collapsing the distinction between kind of politics with all of its messiness and complexity and science, which I think should aspire to a kind of objectivity, which gets lost when we start making these really sort of expansive judgments.

I think we’ll probably circle back on this a few times as we go through this debate. But Sacha, you’re also somebody with very interesting views about not just this question of the kind of prevalence of misinformation, but also about human belief formation and the extent to which, in your view, lots of people, both in popular discourse, but also in academia, kind of overestimate the gullibility of human beings when it comes to exposure to false or misleading content. So do you want to say a little bit about your view concerning human gullibility?

Sacha Altay: Yeah, I just wanted to finish the last point on the fact that, you know, we are criticizing definitions of misinformation, but in media and communication studies for a long time, they have been studying kind of media bias, framing, agenda setting. Like they are very old theories of media, how they can misinform in subtle ways and indirect ways the public. And all of that has kind of been ignored by misinformation research. But now I feel like today misinformation research is catching up and be like, actually, we should go back to these theories. And so I think it’s good. But I just wanted to point that out.

And regarding gullibility, yes, I think it’s quite popular, the idea that people and like large complex events like the Brexit, Donald Trump or whatever are caused by people being irrational or gullible in particular. By gullible, I think what people often mean is that they are too quick to accept communicated information, like social information that they see out there in the world, in the news, communicated by others. And I think that the scientific literature shows something very different.

For instance, there is a whole literature on social learning, so how people learn either from their own experiences, their own beliefs, or what they see compared to like communicated information, social information, advice. And the consensus in this literature is that people underuse social information. They do not overuse it, they underuse it. And they would be better off doing many kinds of tasks if they were listening and weighing other people’s opinion and beliefs more than their own. So, I mean, it makes sense. Basically, we trust ourselves, we trust our intuitions, our experiences much more than that of others.

And so that’s kind of a consensus. There are many kinds of tasks, like you ask people, oh, what’s the distance between Paris and London? It’s like, 300 kilometers. Another participant, say 400. And you’re not going to take into account other people’s advice as much as your own intuition, even though you have no reason to be an expert on this kind of geographical distances. But you still trust yourself more.

And there are also many like theories and mechanisms that have been shown in political communication and media studies that I think suggest that people put a lot of weight on their own priors and their own attitudes when they evaluate and choose what to consume, which greatly reduces any kind of media effects or any kind of outside information. Like people are not randomly exposed to Fox News. They turn on the TV and they select Fox News. And then people selectively accept or reject the information they like the most. And so I think when you take all that into account, like selective exposure, selective acceptance, and egocentric discounting, it complicates a little bit the claim that humans are gullible.

Dan Williams: Yeah, so there’s this sort of popular picture of human beings as credulously accepting, you know, whatever content they stumble across on their TikTok feed. Although when I say human beings, it’s always other human beings, right? This is another point that you make with the third-person effect. Nobody really thinks of themselves as being gullible and easily influenced by false and misleading communication. But when it comes to other people, there’s this kind of intuition which is that, yeah, people are just being kind of brainwashed en masse by their lies and falsehoods and absurdities uttered by politicians and that they’re encountering in their media environment.

And your point is, no, actually, if you look at the empirical research, it doesn’t really support that at all. If anything, people put too much weight on their own kind of intuitions, their own priors, their own experientially grounded beliefs relative to the information that they’re getting from other people. So rather than thinking of many of our sort of epistemic problems as being downstream of gullibility, we should think of in some ways there being the opposite problem of people being too mistrustful, too kind of skeptical of the content that they’re coming across. Is that a fair summary of your perspective?

Sacha Altay: Couldn’t have said it better.

Henry Shevlin: If I can just raise one question here. Reading your brilliant paper, you emphasized, so this is a paper with the Knight Columbia School. You go through all these different misconceptions about how easily influenced people are by different sources, by sort of different peers, by the media, by the news. But this sort of does prompt the question, you know, where do people’s beliefs actually come from?

And you mentioned people’s priors, people’s intuitions, but presumably people aren’t born with these intuitions, they are formed from somewhere through certain kinds of processes. So I’m just curious if you have any sort of thoughts on where do people’s views come from? Because obviously that would suggest, well, that’s the place you go then if you want to influence people, you intervene on whatever is causing this fixation.

Sacha Altay: I mean, my view on beliefs, and I mean, much of my views come from Dan Sperber and Hugo Mercier, who have these theories on like reasoning and the roles of beliefs. And so basically, to answer your question, I think a lot of people’s beliefs are downstream of their incentives and intuitions they have about the world. For instance, vaccines. Vaccines are like profoundly counterintuitive. Like it’s very difficult intuitively to like vaccines. Like first there’s a needle that goes into your arm, there’s a little bit of blood, you think that there is some kind of like pathogens inside the vaccines, like it’s not something that’s very intuitive. So first I would say most, like not necessarily the beliefs, but the attitudes people have about vaccines largely comes from these very general intuitions that they have about contagion, about infections and about all these things.

And then the beliefs, well, people need beliefs to justify their attitudes. And so if your doctor is like, do you want to get vaccinated and you don’t really want to get vaccinated, you can say you’re scared of needles. But if there are also some widely available cultural justifications like, vaccines cause autism, maybe you’re going to jump on it. Maybe you’re not going to jump on it because maybe you’re smart and you know it’s false, et cetera. But you need justifications. And so I think a lot of people’s beliefs comes from this need to rationalize some justifications that they have. And I think that’s also why on many topics, people don’t have that many beliefs because often people don’t really need to justify many of their attitudes. And there’s a lot of work, for instance, in political science on how surveys kind of create beliefs in people because people have intuitions and kind of like vague opinions about all sorts of stuff. But when you ask them, they have to fix it and they have, and in some sense, it creates the beliefs.

So yeah, I would say beliefs mostly come from prior attitudes that people have and incentives that they have to act in the world.

Henry Shevlin: Okay, but those... just to push a little bit harder there, so the prior beliefs, I think we’re just still kicking the can down the road a little bit. Incentives I get. Incentives seem genuine and explanatory here, but presumably it’s not the case that you can predict people’s vaccine attitudes from the degree of phobia they have towards needles, right? Or at least, even if that is predictive, I don’t know if it is, it seems like there’s more going on there. I don’t want to give people, and I think that’s the danger of giving people too much credit for saying, oh, people’s beliefs perfectly track their own incentives. I can totally agree that incentives play a role, but I’m sure just when we think about our own sort of peer groups, right? I disagree with the political views of a lot of my peers, despite us being in the same socioeconomic class, despite us working in the same industry, despite us having, you know, broadly similar interests, I would have thought. So, I don’t know, I can see incentives carry us some of the way, but yeah, they don’t completely close the mystery here.

Sacha Altay: No, of course, of course. I think it’s, you take the example of vaccines. I think most people who get vaccinated, they just get vaccinated because they trust institutions, they trust their doctors. Maybe they have seen their doctors for 20 years, their doctors tell them to get vaccinated, they do it. So that’s the main explanatory agent here is just they trust some institutions, some experts who tell them to do something and they do it.

You wanna jump in, Dan?

Dan Williams: Yeah, I was just going to say, I think it seems like it’s possible to think, and as I understand Sacha, your view, this is your view. It’s possible to think that we overestimate the degree to which people are kind of influenced by whatever content they happen to stumble across in their media environment or the viewpoints that they happen to encounter in their social network—that we tend to think people are too gullible when it comes to those things.

It’s possible to think that, but also to accept that, of course, we are going to be influenced in complex ways by the information we get from people that we trust, from sources that we trust, from our upbringing, from our social reference networks and so on. So the idea that we’re not gullible and not credulous shouldn’t be sort of conflated with the idea that we somehow are born with our entire worldview from the start in ways that aren’t influenced by the media environment and by the testimony that we encounter. Like clearly we’re massively influenced by what we hear from other people, but sort of my understanding of the perspective that you’re outlining Sacha is that process whereby we build up beliefs about the world—firstly, there are some things that just everyone kind of finds natural, like maybe like there’s something weird about vaccines when you hear about the concept, most people just have a kind of instinctive aversion to it, but also things like, you know, my group is good, the other group is bad, or like certain kinds of maybe xenophobic tendencies that come naturally to people and so on. So there are certain ways of viewing the world and certain things which are intuitive, maybe as a consequence of our evolutionary history, and that interacts then in very kind of complex ways with our experiences, with our social identities, with our personality, with the people that we trust, the institutions that we trust, those we mistrust, and so on and so forth. So you can accept all of that and the role of social learning within that whilst also thinking people tend to exaggerate how gullible, how credulous people are when it comes to sort of incidental exposure to communication. Is that your view, Sacha? Is that a kind of accurate representation of it?

Sacha Altay: Yes, yes, yes it is. I think a lot of the reason, like when we change our mind drastically, it’s either because like we have a lot of reasons to trust the source. Like if the BBC says that the Queen died and the BBC says it, the Guardian says it, we’re going to update our beliefs immediately. And most people, even the people who distrust the BBC are going to update their beliefs directly.

And it’s the same if like, I don’t know, my wife tells me that there is no more milk in the fridge and I have to buy some. I’m going to update my beliefs about the milk in the fridge and buy some, you know, in some ways, of course we update our beliefs based on the information that’s provided to us. It’s just that we do so I think in ways that is broadly rational in the sense not that it’s perfect, but that it serves our everyday actions and our incentives, like what we want to do in the world, like very well. So I think that’s also the way in which I mean it is that when we update it and when we do it, we do it quite well, not to discover the truth, but at least to get along in the world.

Dan Williams: And could you maybe say a little bit more about this point concerning selective exposure? So the fact that when people are engaging with media, with the viewpoints of pundits and politicians and so on, a lot of that is, quote unquote, demand driven in the sense that people have strong attitudes, they’ve got strong political, cultural allegiances, they identify with a particular in-group, they want to demonize like those people over there or that kind of institution, et cetera. And it’s these sort of pre-existing attitudes, interests, allegiances, which often build up in complex ways over a long period of time, which then causes people to kind of seek out information and often misinformation, which is consistent with their attitudes and their interests, rather than the picture I think sometimes people have, which is—I think the way Joe Uscinski puts it is, you know, they’re walking along and they slip on a banana peel, you know, they encounter some conspiratorial content on social media and now they believe in QAnon or like Holocaust denial. That’s just not the way that it works. Could you say a little bit more about that concerning like selective exposure and the demand side of misinformation?

Sacha Altay: Yeah, we know for instance that misinformation on social media like Facebook or Twitter, which have been the most studied in particular Twitter, you have a very small percentage of individuals who account for most of the misinformation that is consumed and shared on these platforms. And it’s like very small. It’s like 1% or less than 1% that account for most of the misinformation that is consumed and shared on these platforms.

And these people, they are misinformation sharers and consumers, not because they have like special access to misinformation because they have a lot of money or whatever, but simply because they have some traits that make them more likely to seek out such content, such as having low trust in institutions, being politically polarized. And because of these traits, because they don’t trust institutions, they are looking for counter-narratives to like the mainstream narratives they find on mainstream media. Because the thing is that these people who consume and share most of the misinformation on social media, and give us the impression that there is a lot and that many people believe it—these people are also exposed to mainstream narratives. It’s just that they decide to reject the mainstream narratives and instead of trusting what the TV tells them, they go on some Telegram channels, they go on some weird websites to learn about the world and do their own research.

And this is, I think, some of the strongest evidence, at least in the case of misinformation, that the problem is not in the offer of misinformation because it’s actually quite easy, quite free, quite accessible. It’s super easy to find misinformation online, but most people consume very little of it. But you have a small group of people who are very active and very vocal who consume most of it. And they have low trust in institution and are highly polarized. And I think it matters a lot for how we want to tackle the problem of misinformation. The problem is not that you have a majority of the population that’s kind of gullible and so we should avoid them being exposed to misinformation, rather you have some people who have some very strong motivations to do some specific stuff. And I think we should address these motivations. And because addressing the offer is impossible. And I’m not against like content moderation and stuff. I think we should try to be in an information environment where the quality of the information is the highest possible, et cetera. But if you have motivations to look and to pay or to consume some content, then the offer will be met, like people will create such content.

Dan Williams: Could we maybe just before we move on to these issues about kind of social media and AI, because I really want to get to those, there’s another point connected to this issue about gullibility where I think there’s this massive kind of gap between common sense, conventional wisdom, and what the empirical research shows, which you’ve written a lot about, which is like the impact of things like political influence campaigns and commercial advertising and so on. So you go into that in your paper on generative AI and why you think there’s been a lot of unfounded alarmism about that, which we’re going to get to shortly. But even separate from the issue concerning AI, could you say something about what the evidence that we have actually shows when it comes to the impact of political and sort of economic advertising campaigns?

Sacha Altay: So political scientists have been studying that for a while because in the US there is so much money that is being spent on political advertising, especially in presidential elections. And so the best studies, they come from political science. And to give you an example, some of them have up to 2 million participants that are being exposed to hundreds or thousands of ads for long periods of time, like months.

And so these are the kinds of study that are being done in this field, like very large sample, long periods of time, et cetera. And the consensus is that political advertising in presidential elections in the US has very, very small effects. The effects are not zero because of course, with such big sample size, long periods of time, et cetera, you do find significant effects, but the effects are very, very small, like point of percentage. And so that’s the consensus in political science in the US.

So it’s a bit specific because the US you have like Democrats and Republicans and you socialize these identities and these identities are very hard to change. Like if you’re a Democrat, it’s very hard for you to change and vote Republican. And of course, in the US, you often have only two candidates that are very prominent and people hear about them all the time. So it’s difficult to move the needle. But in like other elections in other country, multiparty, you have more room for political advertisement to have an effect. But even in these cases, even when it’s like lower stakes campaigns with less known candidates, the effects are still quite small. Like, I don’t know why we have this idea that advertisement works very well, it influences people, but at least when it comes to political voting, it’s just very hard to influence people’s vote. And it’s the same for like marketing, like online ads, like on social media—are very ineffective, the thing is that they are very cheap as well. So I don’t want to say that they are useless because they’re actually extremely cheap. So that’s why these companies do them a lot, but they’re also extremely ineffective. And so that’s the consensus in political science.

Henry Shevlin: So I had a question about this in relation to your paper again. It really paints quite a dismal view of the power of advertising in general. And yet this is like a vast global industry. Is it all just founded on sand? Is it all just smoke and mirrors? Are people basically wasting hundreds of billions of dollars a year on advertising that doesn’t, largely doesn’t work?

Sacha Altay: That’s the opinion of many people. Yeah. Many people think that at least it’s overblown. I don’t want to say that it’s completely useless, et cetera. Like, of course, if you want to buy like a washing machine, they all look the same. And if they are all about the same price, if you have more information about one and the information is good and the reviews are good, et cetera, you’re probably going to buy it more. But it’s just you already want to buy the washing machine and you have a price range and you have already like so at the margin, advertisement can work and has an effect, it’s just that the effect, like they calculate basically the elasticity. So how much more when you spend on advertisement, how much more will you sell basically? And the elasticity is like super small. It’s like, I forgot exactly, but it’s like very small.

But yeah, some people have written books about how the whole internet and know, products on the internet, like social media, et cetera, are free because we are the product and they sell us advertisement and stuff. And all of that is a bubble. Some people think that it’s completely a bubble. I don’t think it’s completely a bubble, but clearly I think, yeah, it’s overvalued. I think ads are a little bit overvalued. And I don’t think AI is gonna change that much.

Dan Williams: Okay, so just to sort of summarize what we’ve got to so far. So on this question of how prevalent misinformation is, if you’re focusing on fake news, it doesn’t seem to be anywhere near as widespread as many people think it is. Once you start stretching and expanding that definition to encompass more and more things, yes, misinformation so defined is much more widespread and plausibly is much more impactful, but it becomes so kind of amorphous that it’s difficult to apply scientifically.

Then the second thing we talked about was this issue concerning gullibility, where in your view, Sacha, and I agree with you, even though obviously people are influenced by social learning and there is evidence that, you know, persuasion can work, it can influence what people believe, people also tend to dramatically overestimate how gullible people are.

Let’s now turn to technology and where AI is relevant. And let’s start with social media, kind of very broadly construed. Henry, actually, why don’t I bring you in here? Because I think in a few of our previous conversations, you said something like the following, and you can tell me whether I’m remembering correctly. You said, we can contrast two kinds of cases, like video games and social media. In both cases, there was this big societal panic. Video games are going to make people really violent. They’re going to play Call of Duty, and then they’re going to go out and start shooting people in their community.

And your view is, the evidence there is actually incredibly weak and that there’s very little to support that kind of panic. Whereas when it comes to social media, there was a lot of panic, maybe not initially, actually, I think there was a lot of optimism about social media initially. But these days, there’s a lot of kind of concern about social media and how it’s, you know, destroyed democracy and human civilization itself. It’s this awful thing, having all of this sort of awful set of political consequences. And am I right, Henry, in thinking you’re actually quite sympathetic to that view about social media, even though you’re not sympathetic to the violent video games story.

Henry Shevlin: Yeah, yeah, no, great. I’m glad you bring up this example. Two things. One is I think my main point with that example is about sort of the time course of these worries that with violent video games, we had this massive initial panic that sort of died down as the evidence sort of basically didn’t arrive. As we saw that there wasn’t that as much concern as initially there was we thought there was reason to think there was. Whereas in the case of social media, there really wasn’t that much concern at first. It was seen as, if anything, a positive technology and concern has just sort of grown over time. And that sort of point about the time course of sort of the moral panic is sort of separate from the degree to which these are robust.

That said, I do, I am more sympathetic to the idea that social media presents an array of worries. So I’m probably more sympathetic than both of you to sort of Jonathan Haidt’s worries about the impact of social media and mobile phones on teenage mental health, which is a separate point from misinformation. I also worry about the role of social media and things like political polarization. Again, at least a little bit distinguishable from misinformation. But yeah, I guess I’m a little, at least a little bit worried about the role of social media and misinformation as well.

Dan Williams: Okay, I’ve got sort of views that are difficult to summarize about this. Let’s stay away from the teen mental health, because I think that opens up a whole can of worms, et cetera. Let’s focus on kind of the political impacts of social media broadly construed. Sasha, my understanding of your view is you basically think that the panic over social media and its political impacts is unfounded and it’s not well supported by evidence. Is that fair? Care to elaborate?

Sacha Altay: Yeah. So I’m just going to start by mentioning, I think, the scientific literature and what I think is the best evidence that social media have weaker effects than people think. So there have been many Facebook deactivation studies. So basically, you pay some participants to stop using Facebook for a few weeks. And in the control group, the participants are the same, but they are paid either to stop it for one day or to do something else.

And in general, what these studies find is that when you stop using Facebook for a few weeks, you become slightly less informed about the news and current events, suggesting that using Facebook regularly helps you slightly know about the world and what’s going on in the news. But it also makes you slightly more sad. So you’re slightly less happy when you use social media. So participants who deactivate social media, especially Facebook for a few weeks, are slightly happier. It’s not exactly clear why. It could also be because they are less exposed to news and news is sad and makes people less happy, etc. So it could be that. And there are also many other studies on Instagram.

And basically what all these studies suggest is that the effect of social media on stuff like affective polarization, political attitudes, voting behaviors, is either extremely small or no. And so the effects are very small. But now that I’ve mentioned this literature, I want to mention that there are many critics of this literature and of these experimental designs. For instance, even the longest RCTs are like two months. And of course, two months is super small at the scale of social media. They have been here for years. And you could imagine that it takes a few years for the effects of social media to kick in.

You can also imagine that, of course, participants stop using social media for a few months, but the world continues using social media. People around them continue using social media. So you kind of have these network effects that are possible. And of course, the effects of social media are not individual, they are collective. And so these RCTs are kind of missing the point. They cannot capture the collective and more systemic effects that social media could have. So that’s another critique. And there are many other critiques.

But I still think that what these RCTs show is that social media probably has effects. And there are studies like in collaboration with Meta showing that if you change Facebook or Instagram with like a chronological feed, that is instead of showing users the most engaging content, you show them the most recent content. When you do that, they spend much less time on the platform. Like the time they spend on the platform is diminished by one third.

And it has a lot of effects on in-platform behaviors, but very few effects on out-platform behaviors, on attitudes, on et cetera. So we should take these studies with a grain of salt, but I still think they show us that the effects are probably not as big as at least the most alarmist texts suggest.

Dan Williams: Hmm. I think maybe another critique that some people have raised is that these studies, especially that set of Facebook, Instagram studies that you mentioned, were conducted after there had been a lot of adjustments to the platforms and the algorithms in light of concern about things like misinformation and their effect on polarization and so on.

So that just goes to say, as you say, many people have generated lots of different criticisms of what we can really infer from these studies. I mean, my own view is they tell us something, which is that the most simplistic, alarmist stories about social media don’t seem to be supported by the current state of really kind high-quality empirical research. I don’t think they provide very strong evidence that should cause someone who goes into this with a really strong prior that social media is having all of these catastrophic consequences to update that much. And that then suggests that like how you view this topic is going to be shaped by a lot more than just the empirical research itself. So in your case, I assume that you’ve got these general priors about how media doesn’t have like huge effects on people’s attitudes and behaviors and these things are shaped by all sorts of complex factors other than media. And am I right in thinking that’s doing a lot of the work when it comes to your skeptical assessment over and above these studies themselves?

Sacha Altay: Yes, but I would say the strongest argument maybe in favor of my position is descriptive data on what people do on social media and how often they encounter political content. Because to be politically polarized, you need to be exposed to political content. And there are more and more descriptive studies, some of them on the whole US population in the US, showing that it’s less than 3% of all the things that people see on social media.

So less than 3% of all people see on Facebook is either political or civic content. And there are also super nice recent studies that are using a novel methodology, which is basically recording what people see on their phones. So it’s like a lot of participants download an app and the app records what people see on their phones like every two seconds or so. And these studies have shown that in the last US presidential election, for instance, people have been exposed to content about like Donald Trump less than three seconds per day. So during the US presidential election people have seen so little political content on their smartphones that it’s ridiculous and it’s so small that in my opinion it can only have small effects.

Then again a contrary argument could be it’s the average and they do find that you have a small minority who is exposed to a lot of political information but then again who are these people? Again I think they have attitudes, have priors and they have motivations, they are partisans. And yes, misinformation or content on social media can reinforce, exacerbate, radicalize them a little bit. But I think for the mass and for the general public, who’s generally not that interested in politics, etc. I don’t think it can have very strong effects.

Dan Williams: Yeah, I just want to double click on that and then I’ll bring Henry in. One other kind of stylized fact, which we should flag, which I think is surprising to some people, is if you’re the kind of person who cares about politics and follows the news carefully, and you read political commentary and so on, you are extremely unrepresentative of the average person. Most people don’t follow politics. They don’t follow current affairs closely at all.

And if you ask people very, very basic questions about politics, they are shockingly uninformed about things. That is shocking relative to the perspective of someone like us who follows politics very, very closely. And that’s another thing which I think people who are highly kind of politically engaged often get wrong when they’re thinking about this topic. If the picture in your head when you’re thinking about social media and politics and so on is that the person who’s constantly posting on X about politics is representative of ordinary people. You’ve got an incredibly skewed, misleading picture.

Okay, there’s tons, I think, more to say here. Henry, did you want to come in with any kind of pushback or any more articulation of your perspective?

Henry Shevlin: Yeah, this is all really interesting and helpful. I guess the only thing I’d say is that it seems to me social media has also just changed the kinds of voices that get platformed in the first place in a way that’s both positive and negative. But, we think about things like the rise of Tumblr and its contribution to sort of a lot of so-called, you know, woke discourse, particularly in sort of the late 2010s. And we could equally say the same thing about, for example, reactionary bloggers or neo-reactionary bloggers like Curtis Yarvin and so forth. I think these are the kind of voices that probably just wouldn’t have found an outlet in the prior social media ecosystems. Maybe that doesn’t matter, right? If none of this stuff actually impacts people’s views that much. But it does seem like an interesting shift in our broader political media landscape that social media has changed not just the kind of how much time people spend interacting with content or the way in which they do so, but also the kind of content that gets out there in the first place. Does that figure at all in the impact of these things?

Dan Williams: Sacha, before I bring you on, just want to say just one really quick thing about that, which is the reference to Curtis Yarvin there made me laugh because I think like he’s an example where like the overwhelming majority of people won’t be aware of him. But I think he probably is influential within the kind of ideas, intelligentsia space of the political right. But this idea that like social media and the affordances and incentives of social media kind of changes which voices become influential and prestigious. I think that’s such an interesting and important point, but Henry, I thought you were going in the direction of, like someone like Donald Trump can absolutely murder on social media because he’s so good at like tapping into the attention economy dynamics on social media in a way that, you know, he’ll be much less successful if we lived in a kind of Walter Cronkite kind of media environment.

But there’s this other aspect, which is like the decline of elite gatekeeping, which is characteristic of social media and it’s via that route, I think, where people like Curtis Yarvin can enter the conversation in a way in which they probably wouldn’t have been able to if you go back to like the 90s, 2000s. Sorry, I just had to double click and say that. Sacha, did you want to respond to Henry’s point?

Sacha Altay: No, yeah, I agree. I just also want to say we often mention Trump as like the example of like someone we don’t like who benefits from social media, but there are also people who we like who benefit from social media like Barack Obama. Like he used Facebook a lot during his campaign. He’s super charismatic. And if he was president today or if he was running today, he would do great on TikTok. He still does great on TikTok. Like he’s so charismatic, so good. So it doesn’t always benefit the worst actors.

And I want to say, it’s a very important point about how social media may also shape how politicians communicate. There are some studies, for instance, in France on how short format videos like TikTok is changing how parliamentary members are talking at the parliament. And there are studies showing that especially at the extreme and especially the extreme right, they are doing more and more speeches that are like with more emotions and more, I don’t know, buzzwords. And what they say is that then they post this on social media, and the more buzzwords, the more emotions, and the more all of that, the more it’s going to go viral. And so that their goal is not to convince other parliamentary members, but instead just to buzz on social media and reach some parts of the population.

Then it’s a normative question, whether it’s good or bad. Probably using emotions and stuff is bad. But you could also imagine that if they were speaking to the general public in more authentic way and try to reach them because a lot of people are not interested. That could also be good. But of course, because it’s the extreme right and stuff, we don’t like them. And I think we have good reasons not to like them. But I think we should be careful and we should also think of ways in which it could be used to do good stuff. But I agree that in general, it probably hasn’t done very good. And it’s very hard to quantify it.

Dan Williams: Just before we move on to the topic of generative AI, my view is there’s so much uncertainty in this domain when we’re asking these really broad questions like what’s the impact of social media on politics that we can’t really be very confident about any view that we might have. But it does seem like, at least in my view, a lot of the popular discourse and academic research has focused on things like recommendation algorithms and filter bubbles and so on, where I think I’m very close to your view, Sacha, in thinking that there’s just a lot of kind of unfounded alarmism. But there’s this other aspect of social media, which I think probably has been very consequential, which is just its democratizing consequence. The fact that like prior to the emergence of social media, it was a much more elitist media environment. Whereas now, anyone with a phone, a laptop, whatever, can open up a TikTok account, get on X and start posting about their views.

And I don’t think you need to view that through the lens of, well, that means they’re going to start articulating their views and then persuading large numbers of people. But what I think it does is certain views, which were kind of systematically excluded from the media environment before the emergence of social media, can now become much more normalized. And also people can achieve kind of common knowledge that other people share views that used to be much more marginalized and stigmatized. So those sorts of views can end up being more consequential in politics, even though the views themselves aren’t necessarily more widespread.

And I think you find that with things like conspiracy theories. My understanding of the empirical research, again, people like Joe Uscinski, is that the actual number of people who endorse conspiracy theories hasn’t really increased, but they do seem to play a more consequential role within politics because people with really weird conspiratorial views used to be kind of marginalized in media. Whereas now it’s very easy for them just to start expressing those views online, finding people who share similar kinds of views, coordinating with them. And so it can play a bigger role in politics, even though it’s nothing to do with, you know, mass algorithmically mediated brainwashing or anything like that.

Okay, I’m sort of conscious of time and I really want to focus on generative AI. So there was this big panic about how once we’ve got deepfakes and other features of generative AI, this was going to have really disastrous consequences on elections. It’s going to shift people’s voting intentions in all sorts of dangerous ways. Sacha, you’ve written a paper which we’ve already referred to with Felix Simon. Looking into the evidence on this and presenting a kind of framework for thinking about it, what’s your take?

Sacha Altay: I will start by saying there are three main arguments why people are worried about the effect of generative AI on the information environment. The first one is that generative AI will make it easier to create misinformation and basically to kind of flood the zone with misinformation. The second one is that it will increase the quality of misinformation, better, faster misinformation. And the last one is personalization.

Generative AI will facilitate the personalization of misinformation. I think these are the three main ones and I can go quickly over them and argue why I don’t think they are a big deal. So about quantity, I think that quantity does not really matter. Today we are exposed already to so, I mean, there’s already so much information online and we are exposed to a very tiny fraction of that information. So adding more content does not necessarily mean that people will be more exposed to it. And I think it’s particularly true in the case of misinformation, where I think demand plays a very important role. And so it’s not because there is more misinformation that people will necessarily consume more misinformation. Like it’s not because you have more umbrellas in your store that people will buy more umbrellas. There needs to be like factors, like I don’t know, rain. If it rains more, you will sell more umbrellas. But so there needs to be something, there needs to be like incentives for people to demand more misinformation, to consume more of it. And that’s why I don’t think that the quantity argument is very strong.

I think also the cost of producing misinformation are already extremely low. Like we see it with Donald Trump or whatever, they just say something that is false and they say it with confidence and that’s it. Like the costs are very low. Also, we are very imaginative as a species, like humans have come up with like incredible, fascinating, engaging stories. And of course, AI can improve our innovative skills, but still we are very good at making up stories that make us look good, that make our group look good. And so I don’t think generative AI is going to help that much in creating more misinformation. Regarding quality, yeah...

Dan Williams: Just to interrupt you so that we can sort of take these step by step. So the first worry is generative AI both with kind of large language models and the production of text, but also deepfakes. I take it you’re including kind of both of those categories. The worry is, well, this is going to just really reduce the costs of producing misinformation. Therefore you’ll get this explosion and the quantity of misinformation and that’s going to produce all sorts of negative consequences. And your view is, well, the bottleneck that matters isn’t really quantity anyway. It’s like what people are paying attention to. So you can increase as much misinformation, you can increase the amount of misinformation as much as you want. And in and of itself, that’s unlikely to have a big impact on people’s attitudes and behaviors. Do you have any thoughts about that, Henry, before we move on?

Henry Shevlin: I guess one concern would be even though media environments are flooded with content already, and I completely agree attention is the sparse commodity, maybe you could think of generative media as allowing sort of very niche areas to get flooded with content in a way that wouldn’t have been easy before. I’m just thinking here’s a silly little example, maybe an interesting example from recent media. Some of you may have seen the anti-radicalization game that was launched in the UK, featured this character about two weeks ago, featured this character called Amelia, a purple-haired anti-immigration activist in this fictional game, which was quickly seized upon by a lot of the anti-immigration right in the UK. And now there’s a flood of AI-generated content all about Amelia, mostly making her look really cool and some of it kind of playful, some of it kind of silly. But the point is this was just like a niche news story that I think people found amusing, but I think it would have died a lot quicker had it not been for the ability of people to seize upon this and generate huge swathes of content about Amelia in a very, very short time. So maybe there was just pre-existing demand there, but it would have been demand that would have been perhaps hard to meet without the ability of generative AI tools to create the content to meet that, which maybe is a difference.

Sacha Altay: Yeah, no, I mean, that’s possible. But when you look at the memes on the internet, most of them are like very cheap. It’s just like an image with some text and you just change a little bit the text and we’re probably going to go into that. But it’s the same with like deepfakes. Like cheapfakes are much more popular than deepfakes because they are super easy to do. Like you just change the date or change the location of something and boom you have your cheapfake. And that’s why they are super popular. Yeah, I don’t know, anyway.

Dan Williams: What’s the definition of a cheapfake, Sacha?

Sacha Altay: A cheapfake is just a low-tech manipulation of information. Like you have an image and you change the date of the image, or you change the location of the image. So in opposition to deepfakes, which are like high-tech, completely like for instance, generated image where like it’s usually sophisticated, et cetera. Cheapfakes in opposition, like very cheap, like that you can, most people can do with their computer without like requiring any tech skills basically.

Dan Williams: Sorry, I think I cut you off. I just wanted to give some clarity to people who weren’t familiar with that. Okay, so that’s quantity. And the next thing that you mentioned as a sort of worry that many people have is quality. That is generative AI won’t just enable us to increase the amount of misinformation but increase its quality and initially at least you’re understanding quality as being different from personalization. You’re treating that separately, is that right? Okay, so give us like, surely the concern here is just that, okay, quantity in and of itself isn’t gonna make a difference. But once we’ve got the capacity to generate like incredibly persuasive text-based arguments and deepfakes, even if it’s true that you can create these sorts of cheapfakes and they can be influential, in different contexts, surely the quality of the misinformation must make a big difference to how many people get persuaded by it.

Sacha Altay: Yeah, I think quality is the most perhaps intuitive argument because it’s the idea that you’re going to be able to create images or videos that are unrecognizable from real videos or images. And so of course people are like, how am I going to trust images or videos anymore if they are unrecognizable from real ones? So I think that’s like a very fundamental fear that people have. And I think it makes a lot of sense. It’s very intuitive. But I don’t find it very convincing.

I think it raises a lot of challenges, but I don’t think it raises enough challenges to be alarming. For instance, I think we have had this challenge before with photography. We have been able to manipulate photography in ways that we cannot distinguish them from real photography since the beginning of photography. And how did we solve this problem? Not with technical tools or whatever, but just with social norms about the use of images to not mislead others.

And we have been able to create like fake texts or say false stuff forever and we haven’t solved the problem with like some fancy tech innovation but simply by having rules, reputation, social norms and trusting more or less people based on what they have said before based on what our friends think of them based on their past accuracy and I think all of this we will still be able to use it to help us navigate an environment in which videos could be AI generated or could be real.

And I mean, something I’ve mentioned before, but quite fundamental is that, for instance, we trust the BBC or the New York Times to be broadly accurate most of the time. We also trust them to not use AI in misleading ways and not share like deepfake footages of like presidential candidates that mislead us. And I think this trust and the institutions that exist are sufficient to prevent most of the harm from this.

I think this will have effects. For instance, maybe we will be less able to trust people and sources that we don’t know. Because if we don’t have their track records, how can we trust them that the information they are sharing is true or false or AI generated or not? But I think that’s a very old problem and we will manage. It will make it more complex, but I think we’ll manage. Yeah, Henry?

Henry Shevlin: I was going to say though, isn’t there a worry that sort of new technology creates kind of normative gaps that allow for sort of a kind of annealing or a kind of recalibration of norms? I’m thinking about something here like file sharing, for example. Like I’m of the generation where, you know, Napster, the generation where suddenly it became possible to download music for free. And this created a whole bunch, a whole shift in norms where I think for my generation, at least, you know, this form of theft was basically just completely normalized. Hence we had advertising campaigns like you wouldn’t steal a car, therefore why would you download a song or a movie? And basically pirating went from something that was niche and maybe frowned upon to something that was just completely normalized.

In the same way, I think you might worry that the ease, ubiquity of generative AI is gonna shift our norms around creating fake content. And arguably we’re already seeing this. We had just a very recently the White House itself retweeting pictures, I think of a protestor at an anti-ICE rally and they had manipulated the image, right? And you know, I think if called out on that, they’d probably say, yeah, sure, you know, of course, yeah, we play around with images, you know, that’s what generative AI can do. That’s just the way things work these days, which does seem like a normative shift perhaps, one partly occasioned by technology.

Sacha Altay: My intuition is quite the opposite, is that if anything, challenges that AI, these new challenges of AI will instead increase the epistemic norms that we have. And because we want to know the truth, like we don’t want to be biased. We don’t want to be misled. We don’t want to be misinformed. And so the fact that the challenge is becoming harder, that it’s going to become harder to know if a video is true, authentic or not, is going to make us harder and harsher on people who do as the White House did, where they, we don’t know if it’s them who manipulated it or not, but they share the manipulated images that do not portray her accurately. And so I think people are going to be angry at that. And I think it’s just going to increase how people, the level of the expectation, like what people expect. And I think people are going to expect more. They’re going to expect news outlets and people to be better. I mean, it’s just a prediction. I hope I’m right. I’m an optimist, but...

Dan Williams: So can we connect that to the worry many people have about the liar’s dividend? The idea that, once we’ve got deepfakes, I mean, we’ve currently got technology to create hyper-realistic audio and video recordings, which are basically indistinguishable from reality. There’s the kind of initial worry many people have, which is, my God, people are going to become persuaded en masse that this stuff is true. And I think that’s very unsophisticated as a worry.

But then there’s another story people have which is, okay, maybe it won’t persuade people, but now that you’ve got the capacity to create these deepfakes, politicians, elites, other people who do shady things, they can use the possibility of something being a deepfake to just dismiss any kind of recording which is raised against them as evidence of them doing something shady.

And I guess connected to that as well, there’s the worry people have, which is that just as consumers of content, now if we encounter any kind of audio or video which goes against what we want to believe, we can just say, well, it’s a deepfake. I don’t have to believe it. So we’re just gonna end up becoming like more and more cocooned within our own belief system, not having this access to learn about the world via recordings. So it’s a kind of liar’s dividend worry and this general worry that this is just going to just obliterate the kind of epistemic value, the informational value of recordings. What’s your thought about those kinds of worries?

Sacha Altay: First thing, I think the liar’s dividend does not hinge on AI itself, but rather the willingness of politicians and some elites in particular to lie and to evade accountability and responsibility. AI will certainly be a new weapon in the arsenal and we have seen it in the elections in 2024, etc. Many politicians have used AI to their benefit and many politicians and elites are continuing using them. So for sure, it’s something we should be, we should worry about and we should regulate, et cetera. But will it be a particularly good weapon in the arsenal? Will it be a game changer? I’m not sure. I mean, time will tell. So far, I don’t think it has been particularly good. I don’t think it has been used in particularly good ways. I don’t think people particularly buy it. I don’t think when people share something and then they’re like, no, it was just AI or like try to use AI as the excuse. I don’t think it works very well. And I think there are going to be reputational costs for people who try to do that. We are going to remember that they have tried to do that. And so I don’t know. Again, time will tell. It’s an empirical question. I may be wrong. I don’t know. Yeah, Henry.

Henry Shevlin: I was just going to chime in. I’m sure I’m not alone in having seen on Facebook in particular, lots of cases of AI-generated media being mistaken. I don’t want to pick on boomers too much, but it is often boomers who completely seem to buy it. Like you might have seen these examples of people breaking these glass bridges, these videos that went viral and lots of people, particularly I say older respondents who completely seem to believe this is a real video they’re seeing.

But I guess two responses to that that you might push back with, Sacha, one would be like, well, we’re just in a transitional period, right? This is new. This is so new to a lot of people seeing seeing this concept for first time that they just aren’t aware yet that this is possible and they’ll adjust over time. Another would be to say, look, yeah, maybe if I’m producing a cute image of, I don’t know, a rabbit or an image of someone breaking a bridge or something non-political, it’s easier to convince people that that’s real than it would be the case to, for example, change their political views. So mean, either or both of those responses, things you’d like to go with in response to that.

Sacha Altay: Yeah, I mean, my impression is that if a rando shares a video of Macron doing something crazy, people are not going to believe it. They are going to wait for like France Info and like the real media to cover it. Because if I don’t know, Macron is saying, we are starting a new war with this country. People are not going to believe it, even if it’s very high quality, because they know if it happens, all the media are going to cover it. So I think in very in this high case of like the politician saying something absolutely crazy, people are going to be vigilant and are going to wait for the mainstream media to buy it.

I think many of the AI slop that we see a lot on Facebook, but also on TikTok, are humorous ones. I think there is some part of the boomers, but not just the boomers, who want to be entertained. And for entertainment, they don’t really care whether it’s true or not, whether it’s authentic or not. And you have extremely, you can create extremely cute images of like little animals doing cute stuff and you get what you wanted. You have like this super stimuli, super cute, super entertaining, super engaging. You have what you wanted. Like, and whether it’s authentic or not, I do care. I don’t understand how people don’t, but at the same time it’s like, yeah, it’s brain candy. It’s a candy, brain candy that people get and I don’t see why it’s wrong.

And I just want to point out that we as elites, because we have always looked down on the content that the mass population consumes. Now we look down on like short video formats on TikTok, but we have always looked down on their entertainment practices, et cetera, saying that it makes them stupid, et cetera. And so I think we should be careful about that. Careful about saying that kids are stupid because they are on TikTok and are watching short format video or whatever. I think we should be careful. And I think we are falling a bit into that with the AI slop, but the TikTok AI slop are very different from the Facebook AI slop. The TikTok AI slop are very weird and absurd. And I think they work because they are extremely weird and absurd. You know, they’re something weird about them and people are playing with it. They are playing with the fact that it’s AI and that you can do extremely weird stuff, but it’s very different from the AI slop on Facebook that works, I think, among older populations.

Henry Shevlin: Since we’re discussing TikTok, just a quick point that’s been lurking in the back of a lurking worry I’ve had is it seems to me most of the research focuses on adults. And yet a lot of the worries about both social media misinformation and generative AI misinformation concerns teenagers and young people. And I’m curious, A, whether there’s how much specifically targeted research there is looking at that group. And B, I think there probably are some good prior reasons for worrying about that group more than others, just because teenage years—firstly, our political beliefs are less likely to be stabilised at that point. And secondly, it is obviously an important window for the formation of political identities in the first place. So even if the worries about social media and generative AI misinformation are overblown for adults, could there be more to worry about there in the case of teenagers?

Sacha Altay: No, that’s very possible. That’s a point that has at least has been made for social media and mental health that very few studies have looked at adolescents or young adolescents and that’s probably the group that’s like could be the most sensible to these effects. And so that’s a totally fair point. Regarding generative AI, I also think we should acknowledge that they are also probably much better at using the technology and recognizing it, like whether it is ChatGPT, DALL-E or like all the AI technology, I think they are much better.

And that’s why the AI slop I see on TikTok are like very meta, like they are second degree, third degree, like very meta. Whereas on Facebook, they are just like first degree, like, look, I did this amazing thing. Oh look, this cute baby. So I think very different. So to be honest, I’m not so worried about teens and generative AI on TikTok. Regarding mental health, I don’t know and we need more data, but it’s a very fair point.

Dan Williams: Just on this point about quality, so we’ve been talking about deepfakes, but there’s this other aspect of generative AI, which is just producing kind of tailored text-based content. And there has been this flurry of empirical research, so I’m thinking of like Tom Costello’s work on chatbots and conspiracy theories and so on, work by people like Ben Tappin showing that LLMs can be pretty persuasive with the content that they produce, partly because they’re just very good at recruiting evidence and persuasive rational arguments that is tailored to people’s specific pre-existing beliefs and informational situation. What’s your feeling about the impact of generative AI there? Because presumably there, it’s a very different conversation about deepfakes. And it does seem to me at least that generative AI, you might argue, is going to disproportionately benefit people with sort of bad, misinformed views, because that’s often where you’re lacking kind of human capital, right? You don’t have access on tap to the sophisticated intellectual skills of the intelligentsia when it comes to a lot of this kind of lowbrow misinformation. So they can now access, you know, generative AI, at least if it’s not subject to various sorts of like safety and ethical requirements, and that might happen down the line, isn’t there a real risk there that that’s going to kind of asymmetrically benefit people pushing out misinformed conspiratorial narratives?

Sacha Altay: So it’s good you mentioned these studies because they find super large effect sizes on important topics like politics. But all the authors acknowledge that these effects are estimated in experimental settings and it’s unclear how this would translate outside of experimental settings where LLMs are not going to be prompted to convince participants or users of believing something.

So first, they are not going to be prompted to do that. Second, they are not going to be paid to pay attention and use the LLM in that way. And so that’s why also, you know, Ben Tappin has this piece on like for mass persuasion, it matters more like attention. Are people actually going to do that? Are people actually going to be exposed to that rather than how persuasive it is? And that’s why I’m not so worried.

And it’s important, I think you mentioned the symmetry or asymmetry because I don’t see any good reason why bad actors would be more successful in using generative AI to mislead than good actors using generative AI to inform and make society better or citizens more informed, et cetera. I think in general, good actors have more money, have more trust. Like in France, if the French government releases an AI or whatever to inform people, it’s going to be more successful than if it’s the Russian government. And so in many ways, I think good actors have the advantage, but they need to take it seriously. They need to act and they need to proactively use these tools for democracy and for the better. They should not wait, I think, for the bad actors to attack and them to defend. They should already be using them in the best possible ways to improve society.

Dan Williams: Yeah, my thought concerning asymmetry was just take something like Holocaust denial, right? I think to a first approximation, everyone who believes in Holocaust denial is like stupid for the most part. And if you give them access to highly intelligent generative AI tools, well, they’re gonna be able to use the kind of on-tap intelligence to rationalize that false perspective. Whereas when it comes to the truth, namely that the Holocaust actually happened, we can use generative AI maybe to improve the persuasiveness of the arguments that we’re going to generate, but we’ve already got extremely persuasive evidence and arguments, right? Because that’s where all of the intellectual research and so on exists.

In any case, again, I’m conscious of time. Could we end with this point about personalization? So I still meet people who think that Brexit was due to Cambridge Analytica and micro-targeting and things like this. I think it’s a very kind of common belief people have, which is that once you start targeting personalizing messages, you can have like really huge impact on what people believe. And one of the consequences of AI, very broadly construed, is that they’re gonna greatly enhance the personalization of persuasive messages. So what’s your take on that?

Sacha Altay: Maybe the best evidence is actually the papers by Ben Tappin, Tom Costello and stuff who have actually measured what matters more. Is it whether the arguments generated by the LLMs are targeted to the users based on their political identity, etc., or whether they present more facts and the quality of the facts, etc. And in general, what they find is that what matters is facts. So the more you provide people with facts and good arguments, the more they change their mind. And personalization matters very little.

And in political science, there’s a whole literature showing mostly the same thing, that like, of course you need some targeting, like you need to target based on the language or like some basic level of targeting is needed, but like micro-targeting based on like, yeah, political preferences, values, et cetera, broadly ineffective basically, especially compared to the most convincing arguments you can make.

I think also there is a whole literature in like communication showing that people highly dislike targeted messages when they are very targeted, when they feel like it’s very targeted at them, people recognize it and they dislike it. Yeah, the Cambridge Analytica thing is just a scam basically. I still don’t know why people believe it that much. It’s just a company. They are selling influence. They said they influence major elections and all of a sudden people are like, oh yeah, of course I understand why they do that. People have priors about other people being gullible and being swayed by social media. So when a company said that they sway people on social media, people are receptive to it. They’re not being gullible. It’s just on their priors, et cetera. But yeah, no, there is very little evidence that Cambridge Analytica affected the Brexit or the 2016 US presidential election. And it’s better to present people with good arguments and facts rather than to micro-target them.

Henry Shevlin: If I can squeeze just another angle into the personalization discussion, something you talk about in the paper is relational factors, which is sort of related to personalization, but a bit distinct. And I’m curious about whether you think AI could play a role there. We’ve talked on the show previously about social AI and the idea that young people in particular might be forming deeper and more profound relationships with AI systems or AI friends, companions, lovers, which then potentially could be leveraged for changing their views.

And it seems to me just intuitively that these kind of relations, whether they’re sort of direct relations or more like parasocial relations, can be really influential if we think about, for example, something like Logan Paul’s Prime Energy Drinks. You know, this was an influencer who promoted his own brand of energy drinks that then became a massive sensation, hundreds of millions of dollars, if not billions of dollars in sales over a very short period of time. So it seems like these relationships can be powerful. Is that not a worry that AI could leverage them?

Sacha Altay: And to be honest, I’ve been, it’s very hard, it’s a very hard question. I’m being asked that all the time. And I think the best counter-argument I have at the moment is just, there is very little evidence that people change their mind according to their life partner. Like the people they trust the most, they sleep with, et cetera. There is very little change of mind. And when there is, it’s hard to know whether it’s because the incentives are getting more aligned. Like, you know, they get married, so they are sharing their money, they are buying a house together, they live in the same place, etc. So of course when the incentives are getting closer, you could imagine their beliefs, etc. are getting closer. But basically attitude change is very small with your life partner.

And I imagine that if my wife, who I trust a lot, I love, etc. tells me, GMOs are bad, nuclear energy is bad, etc. Why would she convince me? Like I trust her a lot on many things, but I’m not like completely blind to her. And so how would ChatGPT beat my wife at this, like, I don’t see it, I don’t see it. But to be honest, it’s just my opinion, let’s see how it goes, but I don’t find it very convincing.

Dan Williams: I can confirm that my girlfriend would very much like to influence my political attitudes, but is not having much success as of yet. Okay, one thing we didn’t do actually is you’ve given us your kind of analysis and your belief, Sacha, about the impact or lack of impact of generative AI. But we should mention there were all of these sort of alarmist forecasts about the impact of generative AI and deepfakes on the kind of 2024 election cycle.

And one of the things that you do in your paper is you don’t just go through each individual worry, but you actually kind of survey what the empirical research that we have says. So briefly, what does the research that we have actually say about the impact of generative AI on that election cycle?

Sacha Altay: I mean, to be honest, it’s not like a systematic review, like it’s not super reliable. I just went over and looked at what happened in these elections. And basically, in most countries, the consensus is that there have been some problems with elections, but that it’s old problems with elections, such as politicians lying, trying to gain, to change, like basically politicians doing bad stuff. And generative AI has been used a lot to illustrate what politicians want to say. Often they want to say that they are strong and that their opponent is weak or stupid. So they have been using generative AI to do that in the US, in Argentina, in many countries. They have used generative AI a lot to do some kind of like soft propaganda, portraying themselves and their group as good and the others as bad.

In some countries, apparently, generative AI has been used to do some good stuff, like in India, where we have like many languages and where translation is often a problem and takes time. And apparently, generative AI has been used a lot to translate some political campaigns into all the languages and dialects that exist in India. So I think it’s very varied and not as catastrophic, let’s say, as the alarmist tech suggests. But I think it’s just suggestive evidence. And of course, it’s just the beginning of generative AI. So we should see how generative AI will be used in the future, in future elections. But we should not forget that it can be used to do good stuff. Like it’s not necessarily being used to do bad stuff. You can use it to translate to, and even to illustrate, you can use it to do like faithful imitation, illustration. You don’t need to like portray yourself as super strong and the opponent as bad. You can do, I don’t know, some good or artistic stuff.

Dan Williams: Yeah, we didn’t really talk about the positive side of generative AI very much in this conversation. But my view is, at the moment at least, the kind of boring truth about large language models is that they’re basically just improving people’s access to evidence-based kind of factual information. And I think if you compare the kind of like one-shot answer you get from ChatGPT or Claude or Gemini on any political issue to what you get from the average voter or pundit or politician, it’s just of much higher quality. But I think that truth doesn’t really get the attention that it deserves because it’s sort of boring for the most part. It doesn’t fit into these kind of threat narratives. And it’s kind of counterintuitive because like, why would it be that these, you know, profit-seeking companies that everyone despises have just had a really beneficial consequence on the information environment? But that is in fact, what I think the case is.

Sacha Altay: So you’re totally right because another concern I haven’t mentioned is just hallucinations, like individual users using LLM on their own and being misled by an LLM because they confidently say stuff that is false. But as you say, I think it depends compared to what? How often do they hallucinate and how correct are they compared to alternative sources of information like other human beings, social media, TV?

And I think they would do pretty well actually compared to most of these other sources. And so that’s why I’m not so worried. I think the confidence thing is a bit annoying, but I think most people who use AI regularly kind of know that, yeah, sometimes they completely hallucinate and they go completely awry, but we know it. And I think most people who use it often know it. And that’s why I’m not so worried. But again, it would be better if they did not hallucinate and were perfect, but it’s setting the bar a bit high.

Dan Williams: Okay. Okay, fantastic. Well, thank you, Sacha. We’re going to have to bring you back on at some point because I feel like we’ve just barely scratched the surface with many of these issues. Was there anything that we didn’t ask you that you wished we had asked you?

Sacha Altay: No. I mean, as you said, many things to talk about.

Dan Williams: Okay, fantastic. Well, thanks, Sacha, and we’ll see everyone next time

If you look at humanity, both today and throughout history, you can’t help but notice that people believe a lot of things that seem stupid and irrational. Pick your favourite example: conspiracy theories, religion, prejudice, ideology, pseudoscience, ancestor myths, people who hold different political opinions from your own, and so on.

This observation provokes a central question for the social sciences. Why do broadly rational people, people who often seem intelligent and competent in most aspects of their lives, sometimes believe highly irrational things?

One classic answer is that people are not disinterested truth seekers. In some contexts, our practical interests conflict with the aim of acquiring accurate, evidence-based beliefs. For example, we might want to believe things that make us feel good, that impose a satisfying order and certainty on a complex world, that help us persuade others that we’re noble and impressive, or that win us status and approval from our friends and allies.

Famously, when our goals come into conflict with the pursuit of truth in this way, the truth often loses out. We lie to ourselves, bury our heads in the sand, and engage in elaborate mental gymnastics. Less colloquially, we engage in what psychologists call “motivated cognition”: we—or our minds, at least—direct cognitive processes toward favoured conclusions, not true ones. For example, we instinctively seek out evidence that confirms those conclusions (confirmation bias), shield ourselves from evidence against them (motivated ignorance), insist on higher standards for arguments we dislike than for those we like (biased evaluation), and remember and forget information in convenient patterns (selective forgetting).

Throughout most of history, scholars had little doubt that this tendency was a central and destructive feature of the human condition.

For Adam Smith, for example, it “is the fatal weakness of mankind” and “the source of half the disorders of human life.” For Socrates in the Cratylus, “the worst of all deceptions is self-deception.” And of course, thinkers such as Freud and Nietzsche placed motivated cognition at the centre of their understanding of human psychology.

Against Motivated Cognition

This consensus continued from the emergence of scientific psychology until relatively recently. In the last decade or so, however, some researchers have become increasingly sceptical that motivated cognition is a significant force in human affairs. There are many reasons for this, including reinterpretations of experimental findings, failures to replicate certain findings, and a growing body of evidence that people are broadly rational in how they process information, even in domains like politics.

I’m not very convinced by these sources of scepticism. I think they often rest on naïve assumptions about how to interpret psychological findings and how to understand motivated cognition. When properly understood, motivated cognition is consistent with the finding that people update their beliefs when presented with corrective information.

I also think that, as with human cognition more broadly, most widespread and consequential forms of motivated cognition are distributed and socially scaffolded. They are a “team project” involving complex systems of social norms, incentives, and coordination that function to promote and protect favoured narratives and belief systems. So, if you want to understand how we lie to ourselves, you must move beyond the lone thinker in decontextualised psych experiments and focus on how humans co-construct social worlds optimised for scaffolding self-deception.

There is much more to say about all of this, obviously. But here, I want to focus on a different, more “philosophical” source of scepticism about motivated cognition, one which draws attention to what the philosopher Jeffrey Friedman calls “epistemic complexity”.

Epistemic complexity

“Epistemic complexity” is a bit of jargon for the simple idea that it’s often really hard to figure out what’s true.

Partly, this is because reality itself is often complex, but it’s also due to the highly fallible ways in which we access that reality. We rarely have “direct”, perceptual access to the facts we form beliefs about, especially in domains like politics and religion. Our access is mediated by other people and institutions—priests, teachers, writers, journalists, pundits, scientists, social media feeds, etc.—and by our pre-existing beliefs (“priors”), which, given the world's scale and complexity, typically involve highly selective, low-resolution compressions of reality. Of course, these representations were also primarily acquired from others who are in exactly the same situation.

This is what Walter Lippmann meant when he observed that the modern world is “out of reach, out of sight, and out of mind”, and that public opinion “deals with indirect, unseen, and puzzling facts, and there is nothing obvious about them.”

To make this concrete, consider your beliefs about climate change. Maybe you think it’s our most pressing political problem, an urgent crisis and existential risk, or maybe you think the whole thing is an overblown, leftist moral panic. But whatever you believe, take a moment to reflect on where your beliefs came from.

Reality didn’t just imprint itself directly on your brain, whatever that would mean. You learned about climate change in the same way that you learn about almost everything else: through a highly path-dependent process in which, at every stage of encountering new information (testimony, news reports, articles, education, political commentary, etc.), you filtered it through your priors about the world and about which sources were trustworthy.

Through this process, you arrived at your current opinions, which inevitably take the form of low-resolution compressions of an extremely complex geophysical and political reality into a manageable, understandable form. Indeed, unless you are someone with significant expertise in this area, your “opinions” probably involve little more than socially-learned slogans and soundbites. (To test yourself, open a blank document and write out your current understanding of the topic exclusively from memory.)

It doesn’t take a philosophy PhD to appreciate that this process is highly fallible. Once you realise that the pictures inside people’s heads aren’t simple reflections of reality but the output of complex and fragile processes of interpretation and social learning, you should recognise that there are countless reasons why those pictures might distort or misrepresent that reality.

And yet, most of us don’t intuitively think this way. When we compare our beliefs against the facts, we always find a comforting 1:1 correspondence. Unless we force ourselves to reflect, there doesn’t seem to be a highly fallible process mediating between reality and our representations of it. Reality just is whatever we represent it to be.

The truth often seems obvious, self-evident, so much so that we are frequently baffled when people don’t share our understanding of the truth. The idea that rational people could have encountered the same reality and come away with different opinions doesn’t even register as a serious possibility in many cases. In the language of modern psychology, we are instinctive “naïve realists”. As Karl Popper characterised this intuition, we believe that the truth is “manifest”. If others don’t see the truth, they must, therefore, be deeply irrational, if not outright psychotic.

Given this, epistemic complexity is not merely a feature of our situation that we must grapple with. It is a feature that most people don’t instinctively appreciate, let alone reflect on. That is, it seems much easier to become “informed”—to figure out what’s true—than it really is.

Back to Motivated Cognition

This is where these reflections on epistemic complexity become relevant to questions about motivated cognition.

Historically, scholars have invoked motivated cognition to explain why people hold mistaken beliefs that appear highly irrational. If people confront epistemic complexity, this appearance of irrationality may simply be an illusion produced by naïve realism. That is, once we appreciate that the truth is not self-evident and that it’s extremely challenging to acquire knowledge, we should realise that there is nothing deeply puzzling about why people hold mistaken beliefs. Even perfectly rational individuals will form such beliefs if the challenges of forming accurate ones are sufficiently severe. Perhaps, through no fault of their own, they have simply been exposed to misleading evidence or unreliable sources.

If so, the motivation for positing motivated cognition evaporates. There is no irrationality to explain.

Although this move takes various forms, I think one can find versions of it in the writings of many recent scholars, even when it is not stated explicitly, including Jeffrey Friedman, Neil Levy, C. Thi Nguyen, and Cailin O’Connor and James Owen Weatherall. The core idea is that theorists have traditionally been too quick to jump from observing false beliefs to inferring motivated irrationality. Once we recognise epistemic complexity, we can see that there are countless ways in which individually rational thinkers can acquire false beliefs.

In most cases, these theorists advance alternative explanations that focus on features of the social environment, including how social-informational networks of trust and testimony are corrupted by malicious actors. Hence, this move typically goes hand in hand with the idea that to understand why people hold mistaken beliefs, we should turn our attention away from individual rational failings and toward “structural” and “systemic” pathologies in our society. (Friedman is an exception here, inasmuch as he seems to think that epistemic complexity is so severe that theorists shouldn’t even make judgements about which beliefs are true or false in the first place.)

Motivated Cognition and Epistemic Complexity

It’s an interesting and insightful line of reasoning, but I think it draws the wrong lesson from a recognition of epistemic complexity. Although such complexity opens the possibility that false beliefs can result from rational belief formation, its existence should actually increase our confidence in the likely impact of motivated cognition. This is because epistemic complexity exacerbates motivated cognition, making it easier for us to become convinced of desired conclusions.

In plain terms: The more challenging it is to figure out what’s true, the easier people will find it to lie to themselves.

To see why, think about the factors that determine whether people will engage in motivated cognition. It’s tempting to think that the only relevant variable is the strength of motivations that conflict with the pursuit of truth, such that the stronger those motivations, the greater the propensity to engage in motivated cognition.

However, a moment’s reflection suggests this can’t be the whole story. There are severe limits on what we can convince ourselves of, and these limits are largely independent of the strength of our motives. As Ziva Kunda puts it, “People do not seem to be at liberty to conclude whatever they want to conclude merely because they want to.” One might add: and no matter how much they want to. That is, there is no amount of money (or status, sex, etc.) that could induce me to believe that 2+2=5 or that the moon is made of cheese. These beliefs simply don’t fall within my cognitive grasp.

The reason is simple: For motivated cognition to be possible, we must be capable of providing some justification of the relevant belief. Elsewhere, I have called this a “rationalisation constraint”. But in some cases, we can satisfy it not by explicitly constructing or seeking post hoc rationalisations, but simply by insulating ourselves from disconfirming evidence. (This is captured by the “burying one’s head in the sand” metaphor).

Whatever we call it, however, the point is the same: our ability to become convinced of desired conclusions depends on our ability to feel that they are in some sense justified. That’s why the psychological acrobatics associated with motivated cognition—confirmation bias, biased evaluation, selective forgetting, etc.—are necessary in the first place.

For this reason, the extent to which motivated cognition biases belief depends not only on incentives but also on how easily individuals can satisfy this constraint. To be clear, this isn’t an original point; it’s one of the oldest observations about motivated cognition. The observation that I want to make here, however, is simply that when it comes to justifying desired beliefs, epistemic complexity is a help, not a hindrance. That is, as it becomes increasingly difficult to determine what’s true, it becomes correspondingly easier to convince ourselves of desirable untruths.

This suggests that many people are drawing the wrong lesson from epistemic complexity. Although such complexity implies that even disinterested, rational truth seekers could acquire inaccurate beliefs, its existence should increase our confidence that people are not behaving as disinterested truth seekers.

Of course, it is still ultimately an empirical question to what extent motivated cognition is operative in specific cases. There may be other reasons to think it is less prevalent than many have traditionally assumed. The point is just that the fact of epistemic complexity is not one of them.

So what?

Why does any of this matter? There are potentially many reasons, I think, but I’ll end with two.

First, it’s plausible that elites often benefit when target audiences engage in motivated cognition. So, politicians who spread self-serving lies benefit when their supporters prioritise political tribalism over accuracy. For example, they will be more likely to believe that an election was stolen from their side if they’re motivated to embrace and signal tribal beliefs. This means that many elites have an incentive to do whatever they can to increase a domain’s epistemic complexity—for example, by manufacturing uncertainty, flooding the zone with shit, recruiting congenial “experts”, and so on.

This is a familiar lesson from research on propaganda in some ways, of course, but reflecting on the interactions between motivated cognition and epistemic complexity casts it in a new light.

Second, many studies of the role of motivated cognition in belief formation provide participants with corrective evidence and measure the extent to which they update their beliefs. If they update in a rational direction, this is taken as evidence against the importance of motivated cognition.

One way to understand such experiments is that they artificially and temporarily reduce epistemic complexity. By presenting strong evidence against desired conclusions, they momentarily weaken people’s ability to subjectively justify those conclusions. To the extent that the real-world context in which people think involves much higher levels of epistemic complexity—for example, greater choice over which media and political sources to consult, heightened exposure to conflicting viewpoints and arguments, and greater contact with like-minded friends and colleagues—this suggests that such experiments might be limited in what they can tell us about the real-world significance of motivated cognition.

Chapters

• 00:00 Understanding AGI: A Controversial Concept

• 02:21 The Utility and Limitations of AGI

• 07:10 Defining AGI: Categories and Perspectives

• 12:01 Transformative AI vs. AGI: A Distinction

• 16:15 Generality in AI: Beyond Human Intelligence

• 22:13 Skepticism and Progress in AI Development

• 28:42 The Evolution of LLMs and Their Capabilities

• 30:49 Moravec’s Paradox and Its Implications

• 33:05 The Limits of AI in Creativity and Judgment

• 37:40 Skepticism Towards AGI and Human Intelligence

• 42:54 The Jagged Nature of AI Intelligence

• 47:32 Measuring AI Progress and Its Real-World Impact

• 56:39 Evaluating AI Progress and Benchmarks

• 01:02:22 The Rise of Claude Code and Its Implications

• 01:04:33 Transitioning to a Post-AGI World

• 01:15:15 Predictions for 2026: Capabilities, Economics, and Politics

Transcript

• Please note that this transcript is AI-created and may contain minor mistakes.

How Close Is AGI?

Dan Williams: Welcome back. It’s 2026, a new year, a big year for AI progress, an even bigger year, dare I say it, for this podcast. I’m Dan Williams. I’m back with Henry Shevlin. And today we’re going to be talking about one of the central, most consequential, most controversial concepts in all of AI discourse, which is AGI, artificial general intelligence.

So AGI is written into the mission statements of the leading AI companies. OpenAI, for example, states that their mission is to ensure that artificial general intelligence benefits all of humanity. We also constantly see references to AGI in the media, in science, in philosophy, and in discourse about the dangers, potentially catastrophic dangers, of advanced AI. And yet, there is famously very little consensus on how to even understand this concept, let alone measure our progress towards it.

Is it, for example, a system that achieves something called human level AI? Is it a system that can do any task or at least any intellectual task that a human being can do? Is it a system that performs extremely well on tests, on benchmarks? Or is it, as some people suggest, a deeply confused pseudoscientific concept? So for example, the influential cognitive scientist Alison Gopnik has said, there is no such thing as general intelligence, artificial or natural. Jan LeCun, one of the most famous AI researchers in the world, says this concept makes absolutely no sense.

But if that’s the case, what should we make of people making predictions about when we’re going to reach AGI, perhaps in the next few years? How do we make sense of rapid AI progress? What are we making progress towards? Moreover, what do we make of people, smart people, who claim we’ve already reached AGI, that we’re living through the post-AGI world?

So these are the topics that we’re going to be focusing on today. What is AGI? Is the concept coherent and useful? How do we measure progress towards AGI if we take this concept seriously? And what happens when or if we reach AGI? At the end, Henry and I are also going to be giving some predictions about how we expect AI to develop over the course of this year.

Okay, so to kick things off, Henry, AGI, how do you understand the concept? Are you a fan?

Henry Shevlin: I am a cautious fan of AGI as a concept. I think it’s an imperfect concept and can be very vague or defined in various ways. But at the same time, I think it serves as a useful reminder that we are heading towards an era, in my view, of genuinely transformative capabilities in AI systems. And so when we talk about AI revolutionizing science, AI revolutionizing medicine, AI revolutionizing the future of work, I think AGI is often a useful shorthand for talking about the point at which we start to see really massive changes in these domains.

That said, I do have some sympathy for the worry that this is not a particularly coherent concept. So I think we’ve seen commentary in the media recently saying, look, we don’t really understand what intelligence is, and therefore the very idea of AGI is ill-defined.

What I would say there is that I think we don’t need to understand exactly how human intelligence works in order to recognize when we’ve exceeded human capabilities in certain key ways. And in the same way, we don’t necessarily need to have a perfect biomechanical model of how birds fly in order to build planes that can fly faster than them. So I think even with some empirical questions or some conceptual or definitional disagreements about what intelligence is, what human intelligence is, it could still be the case that we’re well on our way to exceeding the capabilities of human intelligence across the board with AGI.

One thing to quickly flag though is AGI is kind of canonically or classically defined as systems that are equal to human level performance across all domains. I think tacitly this is often restricted to sort of economically and scientifically and cognitively relevant domains, right? So I think if we had systems that were sort of at human level or above in pretty much every cognitive task, but they couldn’t smell or had limited ability to do certain kinds of fine-grained motor tasks, perhaps, I think that wouldn’t disqualify us from characterizing those systems as AGI. If they’re doing better science than a human, if they’re winning mathematics prizes, if they’re Nobel’s, if they’re doing 99% of current jobs in the economy, it’s not going to be a deal breaker whether or not they can tell Sauvignon Blancs from a Chardonnay with a sniff.

Dan Williams: Yeah, although on that point, I think there’s a question here, which is, should we expect a system that can out-compete human beings when it comes to what are thought of as purely cognitive tasks, if it doesn’t have the kinds of competencies that go into, for example, folding laundry, making toast, et cetera. So the idea that you can draw a nice distinction between purely intellectual tasks of the sort that you can perform on a computer, and let’s say what I thought of as sort of non-intellectual tasks, of sensory motor tasks. I think that’s a kind of interesting question in and of itself.

Doing a bit of reading around AGI for this episode, it seems like a lot of the definitions about what AGI is splinter into sort of three different categories. And I’ll be interested to hear what you think about this way of taxonomizing area.

So some people seem to understand AGI basically as a kind of placeholder for whatever AI happens to have really transformative consequences. So it’s like, AGI is just a term for transformative AI, whatever form that transformative AI actually takes. Other people seem to understand it with this concept of human level AI or something similar, where they’re sort of using human intelligence as the thing relative to which we should understand the concept of AGI. And that I think for reasons we can probably get into, I can kind of understand what they’re getting out there, but I think there are all sorts of reasons to be skeptical about that concept. And then there’s a third category of attempts at understanding this concept where you’re just understanding it in terms of kind of abstract capabilities, right? And it might in fact be the case that human beings exhibit or instantiate these capabilities. But the idea is you can specify what these capabilities are independent of thinking about the specific form that they take in human beings. So things like the flexibility and generality of problem solving ability or capacities for continual learning and self-directed learning and autonomy and so on. So it’s like transformative AI understood in terms of impacts, you’ve got kind of human level AI where it’s a system which in some ways has capabilities that are like the sort of ones that human beings have, or you’ve got just a kind of pure capabilities understanding.

Does that correspond with how you’re thinking of this area? Would you add any other categories to that?

Henry Shevlin: Yeah, I think that’s really helpful. I guess a fourth category you might add, it’s a bit of a misnomer to call this category AGI. But I think in practice and a lot of discourse, sometimes people use AGI to refer to something like the singularity or some kind of recursive process of intelligence self-improvement. At which point, AGI functions basically as the same as the idea of artificial super intelligence. I think that’s probably not a maximally helpful way of thinking about AGI. I think it is helpful to distinguish between AGI and sort of the singularity or recursive intelligence explosions. But in practice, that’s what some people mean, I think, they talk about AGI.

Dan Williams: Yeah, just to add a footnote to that. So this idea of an intelligence explosion, roughly speaking, the idea is you’re going to get AI systems that once they can substantially contribute to the process of AI R&D and improving AI systems, you’re going to get this rapid process of recursive self-improvement where every AI system is sort of iteratively involved in building better and better AI systems. We should actually, I think, do a whole separate episode on the intelligence explosion. Because I think reading around about AI, so much of what people are thinking about the future seems to depend on their assessment of like the plausibility of that intelligence explosion concept.

But yeah, so I think you might though think of that as part of that first category of sort of defining AI by its impact in a sense. So AGI by its impact. So there AGI would be, you know, whatever triggers this hypothetical intelligence explosion.

I mean, I think that in addition to what you said, a general problem with defining AGI in terms of the impact of AI, where you’re sort of neutral on what kinds of capabilities might produce that impact, it’s not really forward looking, that kind of definition, right? It’s in a sense, almost by its very nature, going to be backward looking. And it’s not really clear then what we should be searching for or how you would go about measuring AGI solely by looking at the capabilities of the AI systems themselves. So even though I think there is a place for this idea that we need to be thinking seriously about what a world will look like where you’ve got radically transformative AI, merely having this placeholder, to me at least, doesn’t seem that useful as a way of understanding this concept of AGI.

Henry Shevlin: Yeah, I agree. And I think the idea of transformative AI is a useful concept in itself, but I do think it’s worth distinguishing from the kind of more cognitive scientific concept of AGI for a couple of reasons.

The first is that I think you can achieve transformative AGI, sorry, transformative AI, even with quite narrow systems. So there’s this really interesting idea that was very, very central to a lot of AI discourse in the late 2010s called comprehensive AI services. So this is an idea developed by Eric Drexler who said, look, maybe it would be a good idea for safety reasons if rather than trying to build one AI to rule them all, we focus on more narrow domain expert AI systems. So you’ve got an amazing AI scientist, an amazing AI financial analyst, you’ve got an amazing AI writer, but they’re not joined up. They don’t talk to each other at least directly.

And that could be better from a safety perspective, but also pretty much just as useful as AGI. So this is often framed as sort of a choice between two different directions that the future of AI research could go. Part of the problem there is I think LLMs kind of fall between the cracks of AGI and CAIS, as it’s called, Comprehensive AI Services. But insofar as they are a sort of unified system in some sense in terms of their generality, they can do lots and lots of different tasks and they’re not narrow systems. But at the same time, they’re not unified in the sense of being a single psychological agent with memory carried across different instances, capable of coordinating thousands or hundreds of thousands or millions of different conversations towards a single goal. And of course, a lot of the power that LLMs have is their ability increasingly to use various tools rather than sort of having those tools integrated into the systems themselves.

So I think there’s a world in which AI turns out to be transformative that ends up looking a lot more like Eric Drexler’s world. So this is a world without AGI in the sense of, you know, one system to rule them all. Instead, lots of powerful specialized systems, but that still utterly transforms our society and economy. So that’s one reason I think the transformative definition is maybe worth separating out.

Another reason to separate out transformative AI from AGI is something that’s been a big issue in the last year, which is adoption. We could have amazing AI systems or increasingly powerful AI systems, but due to economic or structural factors, they don’t end up at least straight away having the kind of transformative impact that people I think sometimes slightly naively assumed would just happen straight away as soon as you get AGI. So again, you might have, so it’s a sort of a double dissociation. You might have transformative AI that still falls short of AGI because it’s something a bit more like CAIS. Or you might have genuine AGI, but it’s not yet or not immediately transformative because of structural, legal, economic obstacles, things like adoption, to prevent it having the full impact.

Dan Williams: Yeah, and I think that idea that you can’t leap straight from the capabilities of the system to its real world impact is a very important idea in thinking about AI in general. And in fact, we touched on this in our first episode where we looked at the AI as normal technology perspective from Arvind Narayanan and Sayesh Kapoor, where they make a really big deal of this idea that diffusion takes time. There are lots of bottlenecks. There’s going to be lots of risk aversion, need to have all of these other complementary innovations within society in order to actually integrate AI capabilities. I think that’s really important.

Maybe to sort of take a step back. So as I understand where the concept of AGI sort of first comes from, when we separate it from these questions of real world impact and just focus on the capabilities of a system, is if you look at the history of AI, we have lots of very impressive systems, often superhuman, along relatively narrow dimensions, but that could only do some things. So a chess playing system that will destroy the world’s best chess player, but it can’t really do anything else. And even if you just change the rules of chess very slightly, the systems are so brittle that suddenly they’ll lose all of their capabilities.

And one thought was, well, that’s one kind of intelligence, a kind of narrow intelligence, which these AI systems that we were building possessed. But in principle, there could be a kind of intelligence where it’s incredibly flexible and open-ended in terms of the kinds of tasks, the kinds of goals that the system could achieve. And then I take it a question people are gonna have is, okay, why should we expect such a system is even possible?

And a thought many people have is, well, we have human beings, right? And human beings are a kind of existence proof for a certain kind of highly general, flexible, open-ended intelligence, in as much as human beings can become poets, scientists, engineers, dancers, we can play an open-ended set of possible games, and so on and so forth. So the idea is there’s gonna be a kind of conceptual contrast between narrow intelligence and general intelligence. And in a way of addressing skepticism about the possibility of general intelligence, people can always say, human beings have this kind of generality in terms of the sorts of things that we can do.

And I take it that’s partly why so much of the AGI discourse gets translated into human level AI discourse because human beings are supposed to be this kind of existence proof for the kind of intelligence that we’re thinking about. I’m really torn here because I think clearly on the one hand it is true that human beings have a kind of flexible open-ended intelligence that can be combined with an open-ended set of goals and we can perform a variety of different tasks. On the other hand I do really worry about this concept of human level AI, it feels a little bit incoherent to me, like we’re dealing with a kind of great chain of being where there’s this single quantity of intelligence and human beings were on a certain level and we just need to get to that level. That feels a bit confused and sort of dubious to me.

I also think, and actually maybe this is an area where we disagree, ultimately it’s not obvious to me that you’re going to be able to build systems that can do everything that human beings can do that work radically differently from human beings and are subject to a totally different kind of design process in terms of the learning mechanisms by which they arise. I think that idea is coherent, but I think this concept of AGI is basically saying we’re going to get systems that can do everything that human beings can do. They’ve got the kind of flexible, open-ended intelligence, but they’re not going to work anything like how human beings work.

I feel like that idea doesn’t get enough scrutiny in discourse about AI. What do you think?

Henry Shevlin: So loads of juicy threads there. Just a couple of quick historical notes. So the idea of generality as a feature of AI systems was really popularized by John McCarthy all the way back in the sort 60s and 70s, one of the founding figures of modern machine learning and AI. And then I think AGI or the concept of general intelligence as a central notion for frontier model development is sort of popularized and refined a bit by Shane Legge and Marcus Hutter in the early 2000s. So they give this famous definition of general intelligence as the ability to achieve goals across a wide range of environments.

And if we’re going to sort of do any useful sort of scientific analysis, I think, with the concepts in this vicinity, I think the idea of generality as a sort of continuous dimension is more useful and interesting than the concept of AGI per se. I think the AGI sounds like there’s a definite finish line for model development, which I think is probably unlikely for reasons maybe we’ll get onto, but spoiler alert, I think it has to do with the jagged frontier and the jagged nature of AI development. But on the other hand, the idea of generality seems like a really legitimate scientific category, right? Be able to measure, you know, obviously operationalizing these terms is always a bit tricky, but the idea that we can measure the ability of systems to perform well across different domains, that seems like something that is measurable and is meaningful. And I think that’s an area where we’ve seen astonishing progress in very, very recent history.

So back in, I think it was 2019, I wrote a paper with Karina Vould, Matt Crosby and Marta Helena called The Limits of Machine Intelligence, where we were comparing contemporary frontier AI systems somewhat negatively with capabilities, not just of humans, but of non-human animals. In that paper, we draw heavily on biology and just talk about the wide range of things that honeybees can do that birds can do, how they are not specialized intelligences and comparing them with things like AlphaGo or AlphaFold, which are, as you sort of suggested, really, really powerful systems, but operating in very, very narrow domains.

Now, since then, and somewhat, I think, to the surprise of me and others, large language models have shown that in some ways it is possible to build really quite robust systems, systems with a very high degree of generality across a lot of cognitive tasks. And I think that this has sort of dawned quite slowly. I think as recently as sort of just like the launch version of ChatGPT, which was running on 3.5, you still ran into a lot of the kind of familiar problems that you’d run into with sort of previous systems that you alluded to. You change the rules of chess slightly and you get sort of inelegant failures. And I think you could see that already with things like ChatGPT, the launch version would often make non sequiturs. It was easy to confuse. Fairly trivial to get it to hallucinate. And across all those metrics, these systems have been getting more and more reliable.

Early ChatGPT was terrible at mathematics, for example. Contemporary ChatGPT or contemporary LLMs in general can do fantastic mathematics. We’ve had admittedly specialized fine-tuned models, but still LLMs at core that are now winning International Math Olympiad goals. So I think maybe one way to push back against your idea that generality, or at least your hypothesis that maybe generality, high levels of generality, are only achievable in something like a human package. Well, I think the trend line suggests that we are moving rapidly towards more and more general systems in a distinctly unhuman-like package in the form of LLMs.

Dan Williams: Completely agree. And this is, I think, the kind of strongest argument for the alternative view. I mean, just to kind of reconstruct my somewhat garbled reasoning, my thought was something like, we talk about AGI, and often the existence proof that there’s such a thing as AGI is the fact that we’ve got human beings. And I think so much of the discourse about why AGI will be transformative is the idea that these systems will be able to do everything that human beings can do, maybe just in the cognitive, intellectual domains.

And my thought was, well, fair enough, but we’re not building systems that work anything like the human mind, anything like how the human mind works. So there’s a kind of assumption here, sort of bundled with this AGI concept in terms of the way that it gets used, which is we’re going to build systems or we can build systems, maybe we are on track to build systems that can do everything that human beings can do in a way that this concept of AGI sort of captures, but that work nothing like human beings. And I don’t think it’s obvious that that assumption is true. A priori, certainly being a physicalist, being a functionalist doesn’t commit you to the truth of that. So the question is, why should we believe it?

And I think a very good response is look at what’s happening in AI over the past few years. Maybe a kind of skepticism made sense in 2020. But now, just given the realities of how much AI progress there’s been, especially when it comes to the generality of these LLM-based systems, that skepticism is difficult to maintain. I think that’s fair. I definitely think that the progress that we’ve seen in AI and the fact that clearly a significant aspect of this progress is the generality of problem solving ability with these systems. I think that does put pressure on the kind of skepticism that I was raising.

I do wonder how much pressure. Like suppose someone just wants to say, okay, you’ve made a certain kind of progress. We can characterize that in terms of generality. But of course, the people who are really bullish on AI progress, they’re not just claiming that these systems are very competent and general as we find them today, they’re claiming that we’re going to have drop-in workers that can substitute for human labor across different areas of the economy. Why should we extrapolate from progress that we’ve seen over the past four years and think that that’s going to get us to the full suite of capabilities that we associate with human intelligence? We’re kind of skipping ahead here to get to questions about benchmarks and progress and so on. But I think it’s an interesting question. What are your thoughts about that?

Henry Shevlin: Yeah, I think you’ve characterized the debate really well. And I think it was a really plausible hypothesis, even a couple of years ago, that, you know, to use the meme, the phrase that has rapidly become a Twitter meme, you know, deep learning is hitting a wall, LLMs are going to hit a wall, that it was like a really viable empirical hypothesis that we’d find out that there’s only so far you can go with these very unhuman-like architectures. Okay, maybe we find out that you can use them to generate high quality code and do basic composition and translation. But there is some sort of task set T where no matter how big we build the models, they’re just no good. Maybe that would be social cognition or causal reasoning or scientific reasoning.

And yet every candidate domain pretty much has fallen. So I think that doesn’t mean that we won’t find some candidate domains where it turns out these systems just, where just scaling these systems up won’t lead us to greater progress, but we haven’t found them yet.

I think probably the most interesting one that I’m watching at the moment is agency. Some of, I think, I’m not sure if we’ve discussed it before, but things like Anthropic’s experiment with Claudius getting Claude to run vending machines at Anthropic’s offices and failing abysmally any kind of like long term structured planning task that involves interacting with different human agents, some of which might have slightly malicious motives, you know, people trying to get discounts from the vending machine. It’s very funny. We can probably drop a link to the study in the blog. But that was an area where it looks like we really are struggling to build systems that can do something like sustain human agency. But even there, we’re seeing rapid progress. And it’s not clear to me that we’re immediately hitting any sort of brick walls.

So that said, it is entirely possible. And I’d also just emphasize again that I think this is very much an empirical question. I think, again, it was a really plausible hypothesis a few years ago to think that simply training on language alone wouldn’t be able to get you anything like cognition. I think there’s a natural vision of how cognition works where, in the human case, language sort of sits at the top of the pyramid. And then you’ve got layers underneath of things like sensorimotor cognition, motor skills, spatial reasoning, social reasoning and so forth. And language is just the capstone. And if you try and build that capstone without the supporting layers, sure, you might be able to do some clever stuff, but it’s never gonna give you real intelligence.

And I think the discovery that at least that doesn’t seem to be the case from what we’ve seen so far is from just a general cognitive scientific point of view, probably the most astonishing discovery in cog-sci in several decades, I think.

Dan Williams: Can I just quickly interrupt, Henry, because I really want to make sure that I’m understanding what you’re saying. So the last thing you said was you might have a model of kind of agency and intelligence where you need to get the sensorimotor stuff, the kind of embodiment, the being in the world. Is that a Heidegger phrase? I’ve no idea what he meant by that. But that kind of stuff, you need to get that basic sensorimotor stuff, lots of the stuff that we share with other animals, right, first before you can get these more kind of cerebral intellectual tasks like being amazing at software engineering and coding and mathematics and language and so on. And your thought was that was an interesting hypothesis. Actually, what we found with AI in the past few years is it’s not true. Actually, you can get all of that really kind of cerebral, highly intellectualized, those sorts of capabilities without that other stuff.

But couldn’t someone say, well, that sort of cuts both ways in a sense. So we’ve talked about this previously, but there’s this famous, you know, Moravec’s paradox, you know, things that we find easy are hard. Things that we find hard are relatively easy. And that what we found with AI progress over the past several years is, yeah, these systems have got really good with these kind of, we might think of them as evolutionarily recent capabilities that human beings have these very abstract cerebral intellectualized stuff to do with manipulating text and so on. But real, kind of, the significant challenge when it comes to intelligence isn’t that stuff, it’s that sort of basic sensorimotor coordination, these much lower level abilities that we share with other animals. And so far we haven’t seen much progress on those things. And therefore we shouldn’t actually be so bullish on the progress that we’ve seen with these AI systems over the past few years.

Henry Shevlin: Yeah, again, really interesting. I think Moravec’s paradox is looking a lot shakier than it used to. So for it, I mean, one example of something that was sometimes cited as sort of an instance of Moravec’s paradox was image recognition. Image recognition was famously incredibly, incredibly hard, correctly categorizing the kind of things that were in a presented image. And then around 2012, things like AlexNet was one of the early deep learning systems that started to radically tear away these benchmarks and start to dramatically improve on previous generations of performance. And I think it’s fair to say that image categorization is basically a solved problem now.

And I think in quite a few Moravec-type domains, we’ve seen very, very rapid progress. So another Moravec-ish domain is things like understanding conversational implicature or subtle things that people might mean. So conversational implicature, a technical philosophical term, but huge amounts of human language or huge amounts of human communication rely on things like theory of mind and shared context. So if I say, what do you think of that? Whether that is referring back to something I said five minutes ago, being able to figure out what I’m referring to, that’s a very Moravec-style skill that relies on a lot of contextual knowledge. But in these kind of domains, AI just does brilliantly nowadays. AI is very good at conversational pragmatics or conversational implicature, very good at image recognition.

So Moravec’s paradox is no longer, it’s no longer clear that it holds or it holds in a much more uneven and jagged way. It’s not the case that sort of everything that’s easy for a two year old is hard for AI and vice versa. So I think that’s one of the ways in which I would push back.

Regarding the broader question, sure you’ve built sophisticated language models. That doesn’t mean that these systems will then be able to do the fancy sensory motor stuff. I agree. I think that’s absolutely right. So it may not be the case that LLMs five years from now are any better. Well, I think they will be at least a little bit better at the kind of sensory motor stuff as we’re seeing from the increased integration of sort of LLMs into robotic architectures and so forth. But yeah, I think it’s definitely possible that we found a different way to build high-level intellectual capabilities that doesn’t translate to sensorimotor capabilities.

But the other thing I would flag here, and maybe this slightly undermines my own point from earlier on, is that contemporary LLMs are radically different beasts from LLMs three years ago. Contemporary LLMs interpret live video. They interact with the world via querying web results. They can access APIs. They can use tools. They are in a kind of dynamic relationship with the world, albeit one that’s a little bit different from ours. You can ask ChatGPT, is this bar open on a Friday? And it’ll say, yes, I think it is. And say, can you check that? And it’ll come back and say, I’m wrong. Sorry. Yes, they’ve just recently changed their opening hours. They’re now closed on a Friday. I think that is almost a form of sensory motor grounding, you know, in obviously a different package. But so I think contemporary LLMs are, they’re not just sort of these ossified monoliths trained on a bunch of text and then frozen in time forever. They are in some ways closer in, at least at a very abstract architectural level to the kind of dynamic, quasi-embodied systems that we are.

Dan Williams: Interesting. I’m not so sure that Moravec’s paradox has been challenged to the extent that you’re suggesting. I mean, we don’t, we don’t have robotics, right? It’s nowhere near as advanced as these LLM-based systems.

Henry Shevlin: Well, hold on, hold on. Just on that point, what do you think of driverless cars as a counter example here? Because driverless cars were another one of these things that where people often used the failures of driverless cars in the 2010s as an example of Moravec’s paradox in action. They said, these people, actually things like driverless cars are to be far harder than people realize because it involves this whole complex suite of sensory motor capabilities. But now, the safety record of Waymo in the Bay Area exceeds that of human drivers.

Dan Williams: Yeah, very, very good point. I do think though, some degree of goalposts shifting and realizing that certain things we thought would be very hard and much easier than we thought can kind of be legitimate in this context because our intuitions are not particularly reliable when it comes to tracking what really matters about intelligence.

So if you go back to the seventies and eighties, all of these people, even those who thought that embodiment was really central to intelligence, they would say things like, well, you’ll never get an AI system that can beat a human being at chess because that’s going to tap into all of this constellation of abilities, which are, you know, connected to our embodiment and so on. And then obviously we know what happened, but I think part of that is we’re just learning with every kind of development with these AI systems that there’s much more to intelligence than we thought. So yes, we do have self-driving cars, but we don’t have functional robotics of the sort that we can integrate into our lives, suggesting that self-driving cars as impressive as that kind of technology is, is not really a proxy for the kind of full suite of sensory motor abilities that we care about when it comes to animals’ interactions within the world.

I think we’ve also so far been thinking of Moravec’s paradox in terms of this contrast between the highly cerebral intellectual domains, kind of symbolic, often explicitly text-based and basic sensory motor control. But I think there are things like continual learning, right? The capacity of animals, very, very young children, a perfect example of this, to be constantly learning from their environments. And in a way, I think that’s one of these things which state of the art AI today hasn’t cracked. I mean, you’ve got this kind of pre-training phase where it’s next token prediction. Then you’ve got post-training where it’s various sort of reinforcement learning-based learning processes for the most part. But you don’t have kind of continual learning, updating of the model weights as they go through the world from their experience. And that’s not strictly speaking, just a sensorimotor thing. That’s also connected to our sort of higher abilities.

And then also things like, you know, creativity, judgment. We’ve got these words for these concepts. And I think our explicit understanding of them is quite weak. But I do think there’s something to the idea that, you know, ChatGPT, the amount of knowledge this system has is unimaginable relative to what an individual human being has. But individual human beings can do things in the cognitive domain, which is still much more impressive than what systems like ChatGPT or Gemini can do. And again, that’s sort of, it’s not, it’s a kind of competence, it’s a kind of ability, which is not purely sensory motor, but what I think is quite central to how animals in general go through the world, capacity for judgment, for creativity and so on, which again, these systems don’t seem to possess.

And one reason for that might be that they’re just these incredibly weird systems relative to human beings. Their training process is completely different. Their architecture is completely different. And they can do these things that are incredibly impressive, almost unimaginably impressive relative to a few years ago. But there’s a great quote actually from AI podcasting legend, Dwarkesh Patel, which is something like these systems are getting more and more impressive at the rate the short timelines people predict, but more and more useful at the rate the long timelines predict. The thought being, yes, what they can do in terms of our subjective sense of how impressive it is, is amazing. And they’re performing very, very well in terms of these benchmarks. But in terms of their real world utility, actually they’re not having the impact that many people think. And one reason for that might be that they lack many of these kind of amorphous, nebulous capabilities that human beings and indeed to some extent other animals have. Sorry, that was me. I’m not, that was very nebulous and sort of inchoate in terms of their thoughts there, but I’ll be interested to hear what you think.

Henry Shevlin: Well, can I ask, what are some examples of judgment or creativity involving tasks where you think contemporary models clearly fall short of human capabilities? And I’m not denying that there might be such cases, but I’m just curious if there are any ones you have in mind.

Dan Williams: Yeah. Well, for example, I mean, I’m a writer and a researcher. I don’t think AI systems as they exist today, or maybe I should actually, I should rephrase that as commercially released AI systems, because God knows what’s happening privately within these labs. I don’t think they could function as a researcher and as a sort of writer generating novel and interesting opinions, which is the kind of self image that I would like to have. I think they can write bloody well. And I think if you use them as an assistant, it can be incredibly helpful in terms of augmenting and enhancing your abilities. I don’t think we’re at the stage where a ChatGPT could function as a substitute for me, which in a way is strange because it has a knowledge base, which is sort of just so vast relative to my knowledge base or the knowledge base of any other kind of researcher.

So I would imagine if you took my abilities, limited as they are, but combined them with this kind of almost godlike knowledge base of the sort of these systems have, you would get really, really kind of impressive research outputs. But you just don’t see that when it comes to these state of the art AI systems. Am I missing something? Do you disagree?

Henry Shevlin: Well, I think one thing that’s worth mentioning is I think it might be a little misleading to compare you who are, you know, I think, I hope you won’t mind me saying an elite sort of knowledge worker, right, with in thinking about sort of your ability to do original composition, original essays, original analysis. Yeah, I think you still have an edge. But I think we’re well past the point where sort of the median undergraduate essay, I mean, the ChatGPT in its current form can produce far better essays than the median undergraduate essay. I think at this stage, in some domains, it can produce far better essays than the median grad student 5,000 word essay.

And so I think there’s a little bit of a tension there if you’re saying, humans in general have this special sauce that lets us do things that AI systems can’t, when in fact, already AI systems in the kind of domains you mentioned already do vastly better than the very large majority of humans within these tasks.

Dan Williams: Yeah, I think I’m more open to the possibility that they’re doing something very, very weird, incredibly impressive, that does seem to outcompete human beings across specific tasks, but they do in fact lack many properties and capabilities that human beings have such that they couldn’t substitute for them even when it comes to purely intellectual tasks. I do realize though, there is the possibility of a significant amount of copium, self-serving cope in terms of this. And there’s something unsatisfying about it as well, in as much as I think you’re right to, you’re really right to push back. And also I would say, I wouldn’t have predicted the progress of the sort that we’re seeing back in 2020. And I think I probably haven’t fully updated to the extent that a rational individual should have done given the kind of progress that we’ve seen.

But let’s just quickly sort of, let’s return to this, but I’m aware of the fact that we got derailed by a really interesting conversation there. And just take a sort of detour through, we’ve touched on this introductory stuff about kind of AGI. So, you know, how you might understand the concept in terms of transformative impact, in terms of human level AI, in terms of more abstract sort of functional specification of capabilities. Maybe we can just spend a little bit thinking about the skeptical arguments concerning this concept of AGI. So like people like Jan LeCun or Alison Gopnik that I mentioned at the beginning, just saying the concept makes no sense at all and there’s no such thing as general intelligence.

I take it, I mean, one argument you often find here is that human beings are supposed to be the existence proof for the AGI. Here is a, you know, complex information processing system that has the kind of set of capabilities that people that talk about AGI are interested in. But the thought goes, well, human intelligence is not general. The human brain is this integrated mosaic of very specialized abilities that correspond to the kinds of problems we confronted in our kind of evolutionary past.

Sometimes this is cashed out in terms of like massive modularity to get a bit nerdy in terms of the cognitive science debate. And I think people into that kind of perspective, they think there’s something problematic with the concept of AGI because it seems to assume that intelligence is this one generic problem solving ability when in fact human intelligence, which is supposed to be our only existence proof of AGI, doesn’t take that form. It’s this set of special purpose modules for different tasks, which might be nicely integrated in the case of the human brain, but don’t involve just general purpose sort of learning mechanisms. What’s your thought about that kind of critique or that kind of worry?

Henry Shevlin: Yeah, so I’m pretty sympathetic to massive modularity in the human case. I think if you are sympathetic to massive modularity in the human case, that just seems like one way of interpreting that is to say that AGI can operate across, or that general intelligence can operate across highly modular architectures. If what we’re thinking about when we’re thinking about general intelligence is something ultimately grounded in the ability to perform cognitive tasks, right? Does it matter whether that’s achieved purely via a relatively narrow bundle of cells all in your prefrontal cortex or using working memory, or if it’s a bunch of different sort of cognitive sub-modules working together.

So yeah, I think if you accept the massive modularity as a thesis in humans, then why not just say, okay, so maybe the way we get to artificial general intelligence is through a similarly massively modular system. And you can already see hints of this in the way that in the increasing tool use by AI systems.

And it may be that, and this sort of goes back to our discussions about CAIS versus AGI, that the first kind of true AGI systems, I’m skeptical we’ll ever have like a clear, we’ve built AGI moment. But maybe the first systems that sort of get most people would agree are AGI systems might similarly have a relatively modular architecture, maybe with sort of a central coordinator powered by an LLM coupled with a dedicated mathematics engine, coupled with dedicated deep reinforcement learning agents for doing various kinds of scientific work, coupled with, you know, maybe sensory motor systems embedded in drones for doing that kind of thing. I think that would still be AGI, at least in the sense that it’s sort of relevant and interesting.

Dan Williams: Yeah, that’s a very, I think that’s a very good response. I mean, is the worry then that these people have that actually, if you look at AI as it exists today, most of what’s powering it is very general purpose learning mechanisms that doesn’t really look like what you’ve got in the human case. So maybe we should be skeptical that you’re going to get to human-like capabilities via this architecture. But I think your point that actually there’s a lot more kind of modularity here than you might think if you just look at the base model precisely because of this interface with all of these mechanisms. I think that’s important.

I wonder if there’s another kind of thing in the background here, which is skepticism about the way in which AGI often gets talked about where it’s like, we’re gonna build AGI and it’s gonna have these almost superhuman capabilities across all of these different domains. And maybe some people think, well, if you look at human beings, existence proof for this concept of AGI, you don’t find anything like that. You find that we’re very good at some things, we’re not so good at other things. So maybe the thought would be once you’ve paid attention to how human intelligence and maybe more broadly kind of animal intelligence works, very kind of specialized, very modular, that should make you a bit more skeptical maybe about some of the claims about the capabilities of super intelligent AGI in the future. What do you think of that kind of argument?

Henry Shevlin: Yeah, I mean, I think it’s definitely worth stressing how sort of distributed our civilizational capabilities are across different humans, right? I think most humans are not fully general in their intelligence. Some people are great at mathematics, some people are great at coding, some people are great at languages. But we’re able to achieve remarkable things at the civilizational level or at the cultural level because of cooperation across different kinds of specialists within our massive population.

But again, I don’t see why a model like that couldn’t apply to AI systems. Maybe that’s across millions of different instances with different fine tuning to different tasks. So yeah, I think the lack of generality in individual humans is compensated for at the population level. And I don’t see why a similar kind of distributed architecture couldn’t apply to relatively near future AI systems in the kind of modular way I’ve been describing.

There’s an idea here worth bringing back that I touched on earlier on, which is the jagged nature of current AI systems. So for anyone who’s not familiar with this, roughly the idea is if you think about sort of a spider diagram or a radar chart, as it’s sometimes called, where you sort of think about different dimensions of intelligence and sort of map human performance on this, you know, we’ve got sort of spatial reasoning, mathematical reasoning. And let’s just say for the purposes of argument that humans are pretty well rounded across this domain.

AI systems, current AI systems are really, really superhuman already at some tasks, well below human performance on others, around human performance on some. I think this is a really striking observation and a really important observation for understanding trends in current AI. And also explains a lot about the point you made earlier about why these things are maybe less useful than you might have expected.

And, you know, I’ll happily say on the record here that I was far more optimistic about the near term economic impacts of things like ChatGPT, then turned out to be actually correct. If you’d asked me, well, I think I was saying back in November, 2022 on Twitter and places that this is going to revolutionize the economy in the next few years. I still think it is going to revolutionize the economy, but it’s been a lot slower than expected. And I think jaggedness is a big part of the reason, adoption is another.

But just to sort of go into this a little bit more detail, when we think about what an individual human job involves. It involves a huge range of tasks. It’s not one task for the most part. They’re bundled tasks. Current AI systems are really good at some and bad at others, which makes the idea of the drop-in agent-employee model currently non-viable because there are enough tasks within human workflow that AI is really bad at to mean that’s just not applicable.

So a couple of things you might say, how we’re to get around this problem. One is that just rely on these systems getting better and that jaggedness, if not smoothing out, then to sort of the sheer level of the abilities expanding sufficiently that, you know, even if AI systems are still vastly superhuman in some domains and only human level in others, they’ll be good enough across the board that they will be able to function as drop-in agents.

Another interesting idea though is that we will just redesign task flows. We will do some unbundling of tasks in roles such that we create sort of roles that AIs can be dropped in on quite safely. I think a nice useful analogy here, I was talking about this on Twitter not long ago, is if you look at mechanization in agriculture, right, it’s not the case that mechanization in agriculture proceeded through creating robot farmers. It involved instead changing task flows such that relatively simple machines could take over very sort of labor-intensive tasks from humans and changing the kind of things that the average human farmer does.

I think that might be a better model for thinking about at least near-term AI impacts on employment, where it’s a matter of redesigning task flows and value chains such that there are, we do create these niches where you can drop in these AI agents to take on huge important parts of the value chain without necessarily replacing humans one-for-one on the kind of jobs that humans currently have.

Dan Williams: Yeah, that’s really interesting. And I think it’s a very insightful observation. I mean, I would say though, when we’re thinking about what people do in their jobs, it’s not like, you know, there’s a set of tasks that are separate from each other and, you know, AI can do 40% of them or soon it’s going to be able to do 60% of them. The tasks are integrated with each other in an incredibly complex way, such that we might be able to delegate some of these individual tasks to an AI system. But if I think about my job as an academic at a university, it’s not like I can say, my job consists of 142 tasks and here they are. It’s a much more integrated kind of unified set of responsibilities and obligations.

So I think if we’re thinking about not just delegating some tasks to AI systems and adjusting how the workflow is structured and adjusting the structure of organizations, but thinking about radical forms of automation. At the moment, I think that we’re very far from that precisely because I don’t think even as impressive as these AI systems have been, they’re capable of that kind of really kind of long time horizon integrated, like multimodal task performance of the sort that most human beings perform.

And that actually gets us nicely onto something we’ve already touched on, but I think we should think about and talk about as a kind of separate topic, which is measuring progress in AI. So lots of this is framed in terms of progress towards AGI. But I guess you can just think of it in terms of the progress and the capabilities of these systems in general.

So I think there are kind of three overarching ways in which we do this, again, to draw another distinction between three different categories. There’s the kind of subjective, how impressive is this? It’s not completely without value, but I think it is very unreliable for various reasons. There are the sort of set of benchmarks, formal benchmarks that are used to evaluate model performance. And then there is actual kind of real world deployment. So something like what percentage, what fraction of work in the economy is done by automated AI systems or something like that.

If you’re thinking about those three categories, I take it that the quote that I paraphrased from Dwarkesh, where these models are getting more impressive at the rate that the short timelines predict and they’re kind of more useful at the rate that the long timelines people predict. That’s drawing a distinction between two different ways in which you can evaluate these systems. There’s the kind of how subjectively impressive do we find them? And maybe that’s also connected to benchmark performance, where as you’re saying, they’re just getting better and better at a seemingly just ever increasing set of tasks. It’s juxtaposing that with like real world utility. I think that’s complicated, as you said, by the fact that real world deployment is not a simple function of capability, is also going to be shaped by all sorts of other things.

But how are you thinking about this, about measuring AI progress, how to use that to forecast AI progress?

Henry Shevlin: Yeah, I think that’s a fantastic tripartite way of splitting it up. So just a couple of quick comments on sort of the tripartite division. I think as we saw from the launch of GPT-5 last year, how underwhelmed most people were by GPT-5. And I think that that was a fascinating sort sociological episode in itself, particularly because if you look at it purely in terms of benchmarks, the line kept going up and continues to keep going up across most of the things we know how to measure.

There is no evidence of a slowdown in AI capabilities, at least in terms of evals and benchmarks. And yet people were a lot less impressed by GPT-5 than previous models. I think there are a few little just interesting reasons for that. A very basic one is just that cadence of release has massively increased. So it’s no longer the case that we’re waiting a year and a half between model releases with no releases at all in between. Now we get updates pushed every couple of months.

So there are going to be fewer wow moments. I think that sort of partly explains why maybe people were a little bit underwhelmed by GPT-5. Another fact is that I think, another sort of constraint on how impressed people are is that I think models are already good enough at most of the kind of tasks that most people use them for such that new releases don’t radically change people’s affordances with the system.

I mean, I think there are occasional specific domains in which they do. So just to give one example of, I think, a major transition, as it were, in capabilities. I think the release of NanoBanana, Gemini’s integrated image model, dramatically changed what you could do with images and data, specifically because NanoBanana is very good at threading text data and sort of semantic content with images. So for example, with NanoBanana, you can have a long conversation and say, now create an infographic or a mind map of the conversation we’ve just had. NanoBanana from day of release could do that in a way that every other previous image model would fail abysmally at. So there are these kind of like sudden, wow, here’s something new that I could do that I couldn’t do before. But in terms of just sort of general performance of language models, I think they are good enough for most purposes that there are fewer wow moments there.

I think more broadly regarding which of these three ways of measuring AI progress are most relevant and important. I think it depends on the domain. So I think there are isolated domains as measured by particular benchmarks, like some of the mathematics benchmarks, where those benchmarks do have immediate significance, right? If we are using AI models to solve near future, outstanding major problems in mathematics, right? Then I think benchmarks might be getting us pretty close to measuring the underlying criterion that we’re interested in.

Ultimately though, I think it’s the economic impacts that are most pressing and most exciting and most scary. But as you said, there’s so much more to those than just the raw capabilities of the system.

Dan Williams: Yeah. I do think though, I mean, now I’m just becoming a Dwarkesh fanboy, but just to observe another point that he made in this blog post, and we can link to this because I think it was a very interesting one. He says, people who make this point about the difference between, you know, the raw capabilities of the system and the rate of diffusion, or in other words, say that you can’t evaluate the capabilities of the system merely by looking at the degree to which it’s integrated into people’s workflow because that’s going to be slowed down by all sorts of factors. He says that’s basically a cope on the grounds that if we really had AGI, it would integrate incredibly quickly.

And I think an analogy he uses, if you think about, you know, immigrants integrate into the economy very, very quickly because they’ve got this wonderful, flexible, like general purpose intelligence that human beings have. And he says, well, if you really are imagining an AGI of the sort that people like Sam Altman and so on were forecasting, then you wouldn’t have all of this friction when it comes to integrating these systems into an organization’s workflow because they will be able to do everything that a human being can do just better. So it wouldn’t be any more difficult than integrating a human being into it.

It’s an interesting argument. I don’t know whether I’m sort of fully persuaded by it. Before we move on, did you want to respond to it?

Henry Shevlin: Yeah, I think it’s a fantastic argument. I think actually one useful, I think as we move towards greater and greater degrees of generality, the kind of existing structural constraints and the problems imposed by the jaggedness of models are going to become less pronounced. So I think that is a useful way to sort of measure progress towards AGI is thinking about the degree to which systems are capable of overcoming sort of external constraints, external limitations.

So, you know, for example, something as simple as a model being assigned a task, realizing it doesn’t have the internal resources to solve that task, identifying tools it could use so that it could solve that task, and then using those tools. I think that is the kind of behavior. I think we see some of that already with stuff like Claude Code and its current form. But I think that is a good way to think about what progress towards AGI looks like, overcoming the kind of structural constraints that may not be pure limitations of the model, but it has something important about the model if it can work in indirect ways to overcome them.

Dan Williams: Yeah, and I think it’s also interesting because once you start thinking in that way, it really does put pressure on this sort of AI as normal technology perspective that says, look, if you look at the history of technology, the process by which these technologies diffuse throughout the economy and throughout society more broadly, it takes a long time for all sorts of reasons. You might think, okay, but you can’t look at the history of previous technology because something like AGI would be a radically kind of sui generis technology precisely because it will be very easy to very quickly integrate into people’s workflow and into the sorts of things that companies and so on are doing.

Maybe just on this issue of these different ways of evaluating AI progress, before we move on, we could touch on two different things. The first is, so probably the most influential benchmark at the moment is this METR or MATR. I don’t know exactly how you pronounce it. Model Evaluation and Threat Research graph. I think to be honest, we’d have to do a whole separate episode on this where we really get into the weeds because I think the methodology and everything is very, very complicated. But basically, as I understand what METR are doing with this metric, which is basically a time horizon metric, is it saying, look, lots of other benchmarks, they’re evaluating AI’s ability to perform a set of say abstract cognitive intellectual tasks. But what we should really care about or at least one thing we should really care about if we’re interested in these things like agency and the ability to master context and this sort of constellation of abilities that seem to go along with agency is sort of how long it takes a human being who’s a professional to perform a task.

And to the extent that AI systems are getting better and better at performing tasks that would take human beings a very long time to do, that’s telling you something really kind of important about the capabilities of these systems and how fast they’re progressing. And as I understand their metric, basically what they’re saying is there’s been a kind of exponential growth such that the task length of tasks that AI systems can perform is doubling something like between every three or every seven months, where task length there is specified by how long it would take a person to perform that task.

So there’s a very nice quote from Roon, who’s a popular social media AI commentator. He says, the METR graph has become a load bearing institution on which our global stock markets depend. And the thought there is many people are looking at this graph and they’re seeing line go up. They’re seeing this rapid progress. They’re extrapolating that into the future. And that’s why there’s so much optimism about the capabilities of these systems and how they’re likely to develop into the future. That was my current understanding of this graph and the metric that it’s using. Do you have any thoughts about this evaluation?

Henry Shevlin: No, I think you did nothing, no major notes. I think you did a great job of describing it. So just a flag that sort of the METR time horizons task is specifically focused on software engineering tasks. So that is a slightly narrower set of tasks, but it’s one that obviously has massive economic value and is also potentially relevant if we’re thinking about any kind of recursive elements in AI development, you know, software engineering tasks are relevant to building AI systems. So to the extent that we’re finding massive time-saving improvements through the use of AI tools, that might be expected itself to accelerate AI development. So that’s another reason I think that this is so important, maybe less so for the stock markets than sort of the more kind of future-oriented predictions about where AI capabilities are going to go from here.

But of course, absolutely. This is probably the most interesting benchmark to watch when thinking about real world impacts of AI. Software engineers are a very, very expensive group of people to employ. And to the extent that AIs function as massive time savers in those tasks and can do more and more complex workflows within these tasks, that has massive real world impact and significance.

Dan Williams: Yeah, and I think this point about overwhelmingly the tasks are kind of software engineering tasks. So, you know, a software engineering task that might take a human being six hours to complete in and of itself, you know, that’s going to limit the generalizability of this metric because you might think lots of tasks within the world just don’t have the structure of software engineering tasks.

But I think there’s also a sort of, I mean, there are also all sorts of methodological questions about how they’re calculating this and so on. And like I say, I think we should do a separate episode where we sort of dig into this in detail. But I think there’s this other issue which is just to do with benchmarks as a whole. In order for us to be able to have graphs like this, we need tasks in which basically there’s a correct answer or a correct output.

I take it one worry here is just the kind of classic Goodhart’s. Is it Goodhart’s law? You know, when a measure becomes a target, it ceases to be a good measure. So a risk that with any given benchmark, we’re getting systems that are getting better and better at doing well on the test in ways that don’t necessarily correlate with the kinds of things that we really care about.

But I think there’s also another worry where even if you set that aside, the worry would be something like, okay, by the very nature of these benchmarks, where there’s a kind of clearly defined correct answer or output, you’re not tapping into the kinds of things that really matter to a lot of human intelligence, where it’s not a simple issue of here’s the finish line or here’s a clearly defined correct answer or correct output. And I mean, how far do you think that skepticism can go? Like if someone says, look, there’s a possibility here that even though we’re seeing rapid progress when it comes to these benchmarks, including this sort of time horizon benchmarks, which seems like it should be really informative. Nevertheless, it’s just not really telling us anything interesting about the sort of broader set of competencies that matter for real world deployment. Like how much skepticism do you think is tenable when it comes to the gap between benchmarks and sort of the capabilities that we really care about.

Henry Shevlin: Yeah, I think it’s a persistent worry all across not just AI research or even ML in the broader sense, but psychology. Criterion problems is sometimes called, shot all over the place. We have a dozen different ways of measuring creativity, which have minimal predictive validity for one another. As soon as you operationalize a really interesting target, you immediately lose many of the features that make it interesting in the first place. So I think it is an absolute legitimate worry.

That said, I think that I should be able to do better than this, just anecdotally, I think we are seeing models become just generally more useful. If they were improving in benchmarks, but that wasn’t translating into actual real world utility on different tasks, that would be a real red flag.

I can speak about your experience, but my experience is that basically every successive model release is at least somewhat better. I can do some new things with it. And that’s why I think the METR Time Horizons benchmark is a valuable one, but why I also think sort of more grounded economic benchmarks, for example, the degree of internal value created by different, by AI usage in different industries, the degree to which industries are successfully implementing AI automation projects and so forth, they’re an absolutely necessary complement because they’re measuring something that still has some criterion problems, like generating economic value, but is much more tangible and less likely to be a mere artifact of our sort of testing of our evaluation framework.

Dan Williams: Yeah, okay, great. Let’s, I think there are two things to kind of finish on. One of them, I think we can be brief because we’ve really kind of already touched on this, but it’s what we should expect the transition to a post-AGI world to be like, however you understand AGI. And the other is predictions for 2026 in terms of how we see these capabilities.

But first, just want to give you an opportunity to have a take on Claude Code. So, I’m sure you’ve also seen a lot of commentary, a lot of buzz, a lot of discourse to the effect that Claude Code, and just for those who aren’t really in the weeds in AI, Anthropic is a frontier cutting edge AI company. They’ve got a model called Claude. And as part of that, they’ve got Claude Code, which is primarily used for sort of software engineers and coders, but apparently it has much broader application.

I should say I haven’t used Claude Code. I do use Claude all of the time, which I think is incredibly impressive. I haven’t used Claude Code. I’m very, very skeptical that it’s AGI or if it is AGI, I think that probably tells us that the concept of AGI can’t do the work that many people have assumed that it can do. Have you got a take on Claude Code before we move on?

Henry Shevlin: So I haven’t played around with it as much as I would have liked. And it is, I think, one of the more daunting models for non-technical people to use. Even installing it, for many people, will be a little bit of an adventure. But particularly speaking to friends in technical whose jobs are primarily technical, the wow factor seems to be huge on the current iteration of Claude Code.

People are talking about how it’s transforming their workflows, enabling them to do a whole suite of tasks they couldn’t have dreamt of doing before. And I do think it is a significant landmark. Yeah, I think it probably is a taste of the kind of capabilities that we’re gonna see over the course of the rest of this decade, where it’s not just people slotting AI to do specific tasks or sub tasks within their own workflows, but being able to delegate whole workflows to Agile Systems.

Dan Williams: Yeah, okay. And that in a way that leads us onto the first of those two points that I wanted to end on, which is how we should think of the transition to a sort of post-AGI world. I mean, I take it there’s a model you sometimes come across where it’s almost like it’s the atom bomb going off in the Manhattan Project. You reach something called AGI and it’s just radically transformative immediately, for various reasons, maybe because of the capacity to take AGI and use it for large scale automation, but also potentially because of the ability of AGI to get involved in the AI R&D process, triggering this kind of intelligence explosion.

I’m really skeptical that that’s the right way to think about it. I think what we’re seeing basically is kind of incremental improvements in the capabilities of these systems when it comes to things like agency, multi-step sort of long time horizon planning, continual learning and so on. I don’t think there’s gonna be like a big bang. I think we’re gonna see this sort of incremental progress where if you compare, you know, one year to three years down the line, it will seem like this huge disparity, but living through it, I think it’s gonna seem very continuous.

And also when it comes to the impact of this kind of technology on the economy for the reasons that we’ve got into. I think there are going to be all sorts of bottlenecks. There’s going to be so much opposition, even when you’ve got capabilities that are very powerful, integrating it into people’s workflow and so on and so forth. So I’m definitely not really expecting a kind of big bang here. And I think people saying that with highly agentic, at least relative to what’s come before AI like Claude Code, you’re seeing kind of baby AGI, I think that might be true to an extent relative to certain understandings of what AGI is, but that just tells us, I think, that AGI isn’t going to be this landmark event. It’s going to be a sort of continuous incremental improvement across lots of different capabilities. So that’s my high level take. Do you have a different take? Do you have to build on that in any way?

Henry Shevlin: Yeah, I think I largely agree with your take that we’re not going to have a sort Trinity test equivalent moment if we’re going to use the analogy of the Manhattan Project, right? There’s not going to be a sudden moment where a lab says we’ve built AGI. Instead, it’ll seem very incremental and continuous to most people, even those who are following what’s happening. And then by the end of this decade, we’ll look back and say, holy shit, how far we’ve come.

And I think I don’t think there’s any reason to think that progress towards the kind of highly general autonomous systems that, or highly general autonomous capabilities that people associate with AGI. I don’t think there’s any reason to think that that progress isn’t continuing the pace. And I do think, to go back to Claude Code, that it is an example of the kinds of really consequential leaps that we’ll see.

So Ethan Mollick today has a, Ethan Mollick has a piece, brand new piece called Claude Code and What Comes Next, where he talks about using Claude Code to generate a passive income and how it creates hundreds of files for him. Having worked autonomously for 74 minutes, it deploys a functional website that could actually take payments. It got various things wrong along the way, but it was a far cry from sort of the Claudius vending machine experiments from earlier this year, earlier last year.

So yeah, I think we’re going to look back at the end of this decade and realize how far we’ve come, but there’s not going to be a single Trinity test style moment. And I think an interesting parallel here is actually with the Turing test. I think a lot of people were expecting the Turing test to be, there would be like a decisive moment where it’s like, wow, computers can now pass the Turing test. I don’t think that would have been a smart thing to think because Turing’s original test is woefully under-specified. He doesn’t sort of give exact time windows and so forth, and there are various constraints you can build in.

But I think at this point now, the Turing test is no longer especially relevant as a measure of AI capabilities. It’s still of interest, but it’s no longer the case that it’s sort of a clear benchmark we’re working towards. We have had multiple instantiations of the Turing test now that show frontier AI systems can fool humans over two or five minute time horizons with basically at 100% success rate, like where humans are chance at guessing whether they’re talking to an AI system or a fellow human.

But it’s like that benchmark slowly faded into the background rather than being a decisive moment. And I think AGI is going to be very similar. By the end of this decade, I do expect that we will have autonomous, agentic AI systems deployed in pretty much every industry. The vast majority of people’s workflows and daily jobs are going to be very, very different. I don’t think by the end of this decade, for what it’s worth, that we’re going to be looking at mass unemployment.

I actually quite like Noah Smith’s, I don’t fully agree with it, but Noah Smith has this sort of piece on how even in an AGI world, we might still have full employment, leveraging this sort of concept of comparative advantage, the idea that there are always going to be things where it’s cheaper to employ or easier to employ a human to do a given task. And I think that’s going to be one of the things that prevents sort of mass technological unemployment. Also things just like compliance and the fact that you’re going to need to have humans on the loop in many tasks, monitoring AI systems to ensure that you’re abiding by regulations. But I do fully expect the increasingly general AI systems about which will be debates around AGI will seem increasingly irrelevant to be ubiquitous by the end of this decade.

Dan Williams: That’s a nice prediction. Yeah. The prediction concerning AGI is that debates around AGI will go the way of debates about the Turing test. Also, just to add to that point you made about the economics of this, I think the comparative advantage point is very interesting and I think it’s very important.

There’s also a kind of obvious thing which sometimes gets missed in questions about automation, which is when we say, you know, AI systems that can, let’s say, outcompete human beings doing what human beings do. Really, the contrast there is outcompete human beings using AI systems. So it’s not like human intelligence is this fixed target, such that we need to build AI systems that can outcompete human beings as they are in 2026. Human intelligence in general depends on all sorts of technological scaffolding and so on. And that makes it a moving target.

I certainly find in my own work, me with AI, so much more productive and effective than me without it. So if you ask, you know, could AI systems beat Dan without using AI? That’s a very different question, I think, than could you design autonomous, you know, flexible, continual learning based AI systems that could outcompete me with access to those systems? And I think that’s also got sort of implications for how we think about the real world impact of all of this.

Henry Shevlin: Yeah, I guess I do want to, before we go into predictions, I just want to add one final sort of coda here, which is I’ve, in that sort of foregoing prediction of what we’ll see this decade is not to be extrapolated outwards. I think there may well be a point probably beyond the point of this decade where things start to get really, really weird, where sort of the degree of the absolute advantage systems have really just fundamentally starts to reshape workflows and value chains in a way where human labor may eventually, and maybe in some point in 2030s, start to struggle to fit in.

So I’m thinking here of this wonderful piece, very far-sighted flash fiction piece by Ted Chiang called Catching Crumbs from the Table, published in Nature Futures a long time ago, about 20 years ago, where he talks about post-human science and the idea that eventually science reaches the point where it can only be done by AI systems. And just because the kinds of theorems, kind of tools being used are just incomprehensible to humans. And he imagines this sort of cottage industry of sort of explainers where humans try and understand, you know, we’ve built this, AI has developed this new alloy that we are completely incapable of understanding in terms of our existing material science, but let’s do our best, right?

So I think it is possible, I think broadly plausible that if we extrapolate far enough outwards that we might start to hit that point. And then I really do think all bets are off. What does human employment in finance look like when you have super intelligent financial managers supervising super intelligent analysts? What is the role for the human there? Is it just going to be that you sit next to your mainframe running 10,000 super intelligent AI finance agents, and if they ever do anything illegal, you get fired. That might be what sort of people’s jobs start to look like at that point. But I think that’s slightly longer time horizons instead. What I see over the course of this decade is not mass unemployment, but definitely radical changes in human workflows.

Dan Williams: I also think like one of the things that at least in my own case, the reason why I find this so difficult to think about is I just don’t know what to make of the intelligence explosion argument. And I feel like the people that are expecting a real sort of discontinuous leap here, at least relative to human timeframes, they’re imagining a process which will be incredibly rapid precisely because of this model about recursive self-improvement.

So Will MacAskill and Fin Moorhouse have a really nice article on the intelligence explosion and how you could basically compress sort of a hundred years of technological progress into, you know, much, much shorter timeframe. And if that kind of analysis of what you might see with this intelligence explosion, as they understand it, is correct, then, you know, my current view that a lot of this is going to be sort of relatively incremental and continuous and there aren’t going to be any sharp breaks might just break down entirely. I feel like I need to get a good grip on what to think about that whole argument. But we’re going to be devoting episodes this year to people that—

Henry Shevlin: Just to tee up one thought here. So on this point, a sort of prologue or a preview of the future episode, I do think for all of the worries, I think in many cases legitimate worries about hype in Silicon Valley, I think that is absolutely a legitimate cause of concern. And the quasi-religious nature, I think Karen Hao was talking about, you know, how there often is a quasi-religious element to some of these predictions. I think that’s absolutely right. I don’t think that’s disqualifying, right? But I think absolutely, if you don’t think there’s something, there’s a religious element in a lot of talk about AI and AGI in particular, then you’re not paying attention. There absolutely is.

So I think that’s true, but there’s also a bias in the opposite direction, normalcy bias, and the meme version of this is nothing ever happens. But I think if you just look at the recent history of our species, we have many discontinuities, whether that’s the Industrial Revolution, the Agricultural Revolution, even biologically, the emergence of multicellular life in the Ediacaran and the Cambrian explosion, right? The history of life on earth and the history of human civilization is full of these sort of major transitions, these relatively rapid discontinuities. So I think the assumption that sort of nothing ever happens or that, you know, things are basically going to tick on as normal is another bias that we need to be wary of.

Dan Williams: Completely agree. Okay, you said predictions end of decade. How about this year? So when we do this conversation at the beginning of 2027, here’s maybe one way of thinking about it, right? Capability predictions is what I think you expect these systems to be able to do by the end of this year that they can’t do now. Economic predictions. And here I think the central question is, is there a financial bubble here which is going to burst and, you know, potentially even initiate another AI winter, you know, a period in which lots of the enthusiasm dissipates and then maybe, you know, political predictions, right? At the moment, I think people are not aware of what’s coming. People generally are pretty hostile towards AI and pretty, pretty fearful of it, but we haven’t really seen kind of coordinated political movements against AI, where that’s a defining issue. Should we expect to have seen that by the end of the year?

Henry Shevlin: Oh, so many tricky questions. You know the famous quote, it’s hard to make predictions, especially about the future. I want to say in some ways it’s almost harder to make predictions about the near term than the long term, right? Insofar as those predictions have to be more fine grained, more falsifiable. You know, we can say, oh yeah, like by 2030 things will be really different. That’s easy, right? Saying like what’s going to be different by end of 2026 in some ways is harder.

So I don’t expect any massive AI bubbles. I think as the industry matures, like we will hear about various midsize AI companies who’ve been selling vaporware, going bankrupt. And I think the usual suspects will sort of call this out and say, aha, you see there’s an AI bubble all along, but I don’t expect it to be an industry wide trend. I don’t even expect it to be a major bubble in sort of frontier LLMs or frontier model developments.

But the other thing I just emphasise on bubbles when people talk about the AI bubble is that AI is rapidly proliferating into a whole bunch of different things. So like driverless cars, for example, which were, you know, often decried as vaporware in the 2010s, they’re absolutely here now. You can take a Waymo in San Francisco and several other cities today. And 2026 is one of the big years for rollouts of driverless cars. You now have Waymos in London, right? And I think there’s some like 30 cities globally introducing driverless car pilots over the course of this year.

So even if it turns out that OpenAI hit a wall or there’s some major scandal or they’re over leveraged, none of which I think is true. But if that did turn out to be the case, it wouldn’t kill AI in the same way that previous sort of AI winters have sort of killed research in AI, not quite across the board, but more broadly. At this point, everything from driverless cars to autonomous weapons systems to AI and medical research, AI and material sciences to AI for a wide range of tasks. I think it’s too diffuse and too plural for any kind of single bubble events to kill the industry as a whole.

But yeah, I also don’t expect any kind of, even a bubble in the domain of language models. So that’s one point.

Another area where I do think in terms of economic impacts, I think those will grow. I think more and more people are gonna be start, gonna be seeing impacts of AI in their workflows. It wouldn’t surprise me if we start to see some big legacy companies really, really struggling because they’re being outcompeted by startups or scale-ups that make better use of AI than them.

I think more and more companies are going to have to face this difficult challenge of, do we go all in on AI at this point, or do we still try and manage a slow transition? So I think it’s going to be a very economically disruptive year ahead. I think part of the reason for that is I think 2026 really will be the year of agents. So a lot of people, I think Sam Altman said 2025 was going to be the year of agents, but I think that was premature. AI, agentic capabilities of AI understood here as sort of their ability to do long-term complex multi-step tasks is only really getting going. But I think particularly as we start to see more deployment of these AI agents that are in turn generating useful training data about what works and what doesn’t, I think we’ll start seeing more and more really valuable AI agentic products over the course of 2026. I think Claude Code is very much a taste of what’s to come. So big economic disruptions by the end of 2026.

And I think to touch on your political point, I think this is going to lead to increasing backlash. A really interesting phenomenon at the moment is that on the right in America, there is a relatively unified or at least superficially unified pro-AI mood. I think a lot of this has to do with the influence of, you know, a lot of big or the alignment of a lot of big tech with the Trump administration, which has its own reasons for being very pro-AI, geopolitical considerations and so forth. But I think one interesting prediction would be that the right wing on the American right in particular, maybe the global right, pro-AI attitude may start to break down.

I think we’re seeing some trends, some signs of this in domains like AI in young people. There’s an increasing number of sort of Republican politicians who are very, very concerned about things like LLM psychosis, about appropriateness of content that minors are accessing, about impact on youth mental health.

And I think we might actually start to see, that’s one of the areas where we might start to see in the US context, some bipartisan consensus emerging on the need for AI regulation. I’d say partly that’s maybe due to things like family values, protecting young people being values that are as central, if not more central to the right than on the left. So it’s inherently bipartisan, but also because the idea of better regulations around protecting young people don’t necessarily interfere with kind of geopolitical applications of AI. You know, strict rules on under 18s using ChatGPT is not going to prevent the US from using AI tools effectively in future military conflicts and so forth. So that’s one political development.

I think at the cultural level, things are just going to get weirder and weirder. So, you know, we’ve done two episodes on social AI. I think 2026 social AI is going to continue to become more and more ubiquitous.

Sadly, I think we will see many more New York Times and legacy media stories about LLM psychosis, LLM exacerbated or triggered or implicated suicides. I think we’re going to continue to see deep entanglements, deep relationships between humans and AI systems become more and more common. And maybe an outside prediction, I do think the kind of AI welfare, robot rights movement is going to continue to gather steam. Probably not a major culture wars issue, even by sort of this time next year. But I think it’ll, you know, it will go from being this, I mean, it’s no longer even that niche, but still relatively niche thing worked on by a few think tanks to something that is increasingly something the general public are thinking about.

Dan Williams: Great stuff, great stuff. I think many of my predictions, to be honest, overlap with yours and probably a unifying theme is I expect all of these things to happen kind of gradually. So I think the systems will get better and better, not just in terms of how they perform on benchmarks, but in terms of their capabilities. But I don’t think we’re going to be seeing like a big bang upgrade this year.

On the stock market, I mean, there I think there might be a financial bubble that bursts, even though I take your point that AI itself as a technology is not going away. And I think people often conflate those two things in an important sense that they’re kind of orthogonal in the sense that it could be the case, and I think it definitely will be the case, that AI becomes increasingly impressive, capable, and integrated into the economy and into society more broadly. It could also be the case that given the financials of many of these companies and investment decisions, et cetera, et cetera, that you see some quite significant bursting of the bubble that has short-term significant economic impact. I’m probably kind of 50-50 on that, and I just don’t feel like I’ve got the expertise to really evaluate it.

All of the other things, I think you’re sort of directionally correct as they say. I do think one thing where again, I’m probably 50-50 is my understanding of all of the big AI sort of frontier labs at the moment is a major focus is on this kind of continual learning, building advanced AI systems of the sort that we’ve got today that can engage in continual kind of experience-based learning. So you don’t have to construct kind of bespoke reinforcement learning environments for specific tasks, but you can drop a system into an environment and it will be able to update its weights sort of continuously as it engages with that environment in much the same way that human beings and other animals can do.

Given that, at least to me as an observer, it seems like there’s so much focus on that and a recognition that that will be a really big change. I suspect that this time next year, maybe I’m sort of 50-50 here, that we will have seen at least one AI lab that’s made some significant progress on that. I don’t think it will be kind of immediate now they can do it, but I think maybe there’ll be a paper that’s released. Maybe there’ll be a kind of updated model that can do some version of this. And that would be a really huge, I think, story in terms of the historical development of these technologies. Other than that, I think I basically just agree with you. Directionally—

Henry Shevlin: Yeah, directionally correct is the best kind of correct. I know, yeah, I think there’s only not too much disagreement there between us. I mean, just to throw in one final thought, I wouldn’t be surprised, despite everything we’ve said, if AI is not the biggest story of this year. I think we live in an exceptionally unstable time, probably the most unstable time of my entire lifetime. And I wouldn’t surprise me at all if geopolitics or, I mean, particularly geopolitics, but potentially other domains create bigger surprises that swamp the relevance of AI, whether that’s war in the South China Sea, a major break between Europe and the US.

And that, you know, we’re focused here on AI, but I think that will have very big implications potentially for AI just because AI supply chains are so delicate. You know, a war in the South China Sea, for example, I think could be one of the biggest spoilers for most people’s AI timelines. So despite all my excitement around AI, I think given the sheer instability in the world right now, it may not end up being the biggest story of 2026.

Dan Williams: We are cursed to live in interesting times. Okay, that was such a fun conversation. We’ll see everyone in a couple of weeks.

I started this blog on January 1^st, 2024, so I’ve now been publishing weekly essays here for over two years. It was one of the best decisions I’ve ever made. I’m grateful to everyone who reads and engages. Even the haters and losers (of which, happily, there aren’t many) often provide interesting and informative critiques.

I’m especially thankful to those who have paid subscriptions. I’m aware that many of you subscribe not simply to access paywalled articles but to support my writing. I’m truly moved by this. It’s also a helpful corrective to my broadly cynical views about human nature.

As of 5th January 2026, the blog has roughly 19,800 subscribers. It averages approximately 120,000 views per month, though with substantial variance.

In this post, I will review the blog’s output from 2025, recommend the best things I read last year (as well as other favourites), and then briefly outline how I will approach this newsletter in 2026.

Year in Review

Based on the number of readers, here were my top ten essays in 2025:

1. Status, Class, and The Crisis of Expertise — This argues that one underappreciated factor driving the “crisis of expertise”, and hostility towards knowledge-producing institutions more broadly, is feelings of humiliation and resentment among conservative voters with low levels of education, who view experts as a condescending and hostile social class. Among many others, it draws on the work of Thorstein Veblen (whose concept of conspicuous consumption inspires this blog’s title), Marcel Mauss, Will Storr, Musa al-Gharbi, David Hopkins, and Matt Grossman.

2. Let’s Not Bring Back The Gatekeepers — This argues that the media transformations of the digital age have created new pressures and responsibilities for small “l” liberals like me. Put simply, if you can no longer control the public conversation, you must participate in it, which, especially in recent years, too many liberals have been unwilling to do.

3. Is Social Media Destroying Democracy—Or Giving It To Us Good And Hard? — Much of the discourse about how social media is terrible blames engagement-maximising algorithms. Because companies profit by keeping people engaged and glued to their screens, algorithms feed people the epistemic equivalent of junk food: content that generates outrage and resentment, inflames our tribal instincts, and taps into negativity bias. Although important, I argue that a bigger factor is simply that social media has radically democratised media. Many people have ugly, illiberal, misinformed, and generally bad views and values, and social media gives them a platform and much greater consumer power. Admittedly, this view is not very politically correct to acknowledge, but it’s accurate.

4. On Highbrow Misinformation — There’s a tendency to think that “misinformation” is entirely something that right-wing elites, sinister corporations, and uneducated hoi polloi engage in. But in fact, there is a considerable amount of left-coded “highbrow misinformation” that circulates within the prestigious knowledge-producing institutions that bang on about the dangers of misinformation. I give many examples in this essay and also explain why and how such misleading content emerges and proliferates, often as a consequence of the politicisation and progressive groupthink that has captured many institutions.

5. The Case Against Social Media is Weaker Than You Think — This essay summarises and develops ideas from an article I wrote for Asterisk magazine (“Scapegoating the Algorithm”). The main point I make is that although social media platforms obviously aren’t harmless (see Essays 3 and 4), most of the discourse surrounding their dangers is driven more by vibes, anecdotes, and moral panic than rigorous argument or social science.

6. The “Everyone is Biased” Bias — This essay makes the simple point that although everyone is biased in ways that are important and under-appreciated, it’s not the case that everyone is equally biased. There are significant differences between individuals, norm-governed communities, and institutions in how they handle and process information. So, a recognition of the universality of bias must co-exist with avoidance of the “everyone is biased” bias, which flattens such important differences.

7. The World Outside and The Pictures in Our Heads — This provides an opinionated summary of the Lippmann–Dewey debate over democracy, public opinion, and the role of experts in complex, modern societies. I am a huge Walter Lippmann fan. I think he’s the most insightful political epistemologist of all time. This essay sets out his views on the essentially insurmountable challenges of acquiring adequate political knowledge and understanding in the modern world.

8. On Conspiracy Theories of Ignorance — This essay explores Karl Popper’s critique of the “conspiracy theory of ignorance,” which assumes that the truth is so self-evident that popular false beliefs must result from some deliberate conspiracy. Although Popper was mostly concerned with how Marxists and other leftist intellectuals think about “ideology”, the critique is equally pressing for much establishment hysteria about “disinformation” and “merchants of doubt” as the source of all popular misperceptions. I try to explain why Popper’s critique is valuable even though the world does in fact contain highly consequential conspiracy theories of ignorance.

9. On Becoming Less Left-Wing (Part 2) — This is the second in my series of essays detailing how I have become less left-wing in recent years. I explain in greater depth than I have elsewhere why political knowledge is, in general, extremely hard to attain, how tribal allegiances and other interests inevitably distort our beliefs, and why political ideologies are both inevitable and inevitably simplistic, selective, and vulnerable to distinctive failure modes. Think of it as “postmodernism but good”.

10. Domination and Reputation Management — A popular theory of “system-justifying ideologies”—for example, the belief in the divine right of kings, or that group-based domination is legitimate because subordinate groups are intellectually and morally deficient—is that they function to persuade the oppressed to acquiesce in their oppression. I argue that the real function of such ideologies lies in reputation management among oppressors. This leads me to a broader account of how reputation management doesn’t just produce apologetics for power; it also distorts the belief systems of those who think they’re “unmasking” power, including many “radical” left-wing intellectuals whose critiques of “ideology” were easily co-opted by history’s most despotic, exploitative regimes.

There are several unifying ideas across these essays:

• The truth is not self-evident, even though we are often disposed to think that it is. Reality is vast and complex, much more complex than we can even imagine, and we access it not directly but through messy, often-opaque chains of testimony, trust, categorisation, and interpretation. Even the part of reality that we are in “direct” contact with—the bits we can actually perceive—are typically understood through socially-learned conceptual schemes and belief systems. As Walter Lippmann put it, modern politics deals with “indirect, unseen, and puzzling facts, and there is nothing obvious about them.”

• Experts are necessary but human. Although journalists, pundits, intellectuals, scientists, and other “epistemic elites” have critical advantages in confronting and uncovering such facts, they are also vulnerable to the same biases as everyone else. Moreover, their advantages are often used to indulge such biases rather than correct them. The critical theorist who “unmasks” ideology doesn’t escape ideology. “Misinformation experts” aren’t strangers to misinformation. And so on.

• The epistemic is not merely epistemic. The beliefs, narratives, ideologies, and social norms that regulate our minds and behaviour are distorted by propaganda, grubby motives (e.g., self-interest, reputation management, and status competition), and tribal allegiances. Such distortions are obvious in our rivals and enemies but not in our friends or ourselves. The failure to correct for this bias produces lots of bad social theory and politics.

• Humans are kinda sorta rational. The popular image of human beings as credulous fools riddled with cognitive biases is mistaken. We are far from perfectly rational, of course, but people—yes, even the people you dislike—are typically far more sophisticated, critical, and intelligent than they seem. The contrary impression arises from a combination of the “third-person effect”, misunderstanding people’s real goals (e.g., assuming their primary motivation is always to figure out the truth), and underestimating the challenges of acquiring knowledge in complex, modern societies (see above).

• Avoid technopanics. Technology is highly consequential, but most popular (and much scholarly) discourse about technology involves simplistic moral panics that obscure the complex, sophisticated ways people use such technologies, and their interaction with pre-existing features of societies. People aren’t passive, credulous victims of algorithms. And the effects of social media platforms are often mediated by long-standing pathologies of democracy, public opinion, polarisation, the growing diploma divide, and the politicisation of institutions, many of which are far more complex and uncomfortable to discuss than algorithms and Russian bots.

Podcasting

I appeared on several podcasts this year, including Evolutionary Psychology and The Good Fight with Yascha Mounk. Both provided really valuable outlets for discussing my views about human nature, belief, and self-deception (in the former) and misinformation, institutions, social epistemology, and politics (in the latter).

In the last few months of the year, I also started an AI podcast with my friend, Henry Shevlin, where we discuss the big-picture philosophical, scientific, and political questions thrown up by rapid developments in artificial intelligence.

I am convinced that AI will be utterly transformative in the coming years and decades. Although I did my PhD (between 2015 and 2018) on various philosophical questions surrounding generative AI, I immediately pivoted to the area of “political epistemology” in the years that followed, albeit still with a strong focus on psychology and cognitive science in a way that distinguishes me from most scholars in this area.

Now, I am back to thinking about AI a lot, focusing less on the technology itself than on its social and political significance (including its interaction with questions concerning misinformation, institutional trust, expertise, and public opinion).

My podcast with Henry is a way to keep up to date with this area in ways that other people will hopefully find beneficial. In the first six episodes, we covered big-picture debates about AI and existential risk, consciousness, education, LLMs’ environmental impact, and relationships:

1. AI Sessions #1: AI – A Normal Technology or a Superintelligent Alien Species?

2. AI Sessions #2: Artificial Intelligence and Consciousness – A Deep Dive

3. AI Sessions #3: The Truth About AI and the Environment

4. AI Sessions #4: The Social AI Revolution – Friendship, Romance, and the Future of Human Connection

5. AI Sessions #5: How AI Broke Education

6. AI Sessions #6: AI Companions and Consciousness

I will always be a writer first and foremost—that’s where my strengths lie—but I’ve found these conversations to be really enjoyable and stimulating. This year, we will be speaking to many interesting guests.

Recommendations

2025 was an excellent year for Substack. I spend more time reading articles on this platform than on any other. For those (like me) interested in science, philosophy, and intellectually serious, evidence-based contributions to politics and current affairs, there is nowhere better.

I am reluctant to name specific Substackers I enjoy because I know I’ll accidentally leave out many brilliant ones. But if you want suggestions on whom to read, you can check my Recommendations, and also follow me on Notes in the Substack app, where I do my best every day to share the excellent articles I come across.

I read fewer new books than usual this year. The main reason is that I’ve been re-reading extensively with LLMs such as ChatGPT, Claude, and Gemini.

Book quality is extremely heavy-tailed. Most books are bad. A tiny number are exceptional. So, the overall value you get from reading is heavily influenced by decisions about what to read, and you are often much better off trying to master and internalise the ideas of exceptional books than reading new ones.

LLMs make this a lot easier. You can upload a PDF of the book and have a quasi-conversation with it, testing your understanding, receiving tailored explanations and tutoring, creating flashcards to import into programs like Anki (for spaced-repetition-based learning), and more. If you haven’t played around with NotebookLM yet, you’re making a huge mistake. So, this year, I spent much of the time I would have ordinarily spent reading new books on implementing this process for the best books I’ve already read.

Nevertheless, I did read some new books. More precisely, I read several new and several old books for the first time. In no particular order, here were the best ones:

Read more

The philosopher Jason Stanley has recently “fled” the USA, which he views as an authoritarian state undergoing a coup by a fascist party using Nazi tactics. In an interview outlining this perspective, he has harsh words for those who draw on the concept of “polarisation” to understand these developments:

“All the people talking about polarization are just fascism enablers. They’re almost worse than the fascists because they’re just like, “Hey, how do I keep getting money in power?” I’ll say the fascists are normal.”

Talk of polarisation is so objectionable, he says,

“because one side is led by fascists. I mean, it’s like saying the… problem with the Civil War was polarization. It’s literally like that… One group thinks that slavery is good, and the other group thinks it’s bad, terribly polarized. Or Nazi Germany. One group thinks Jews should be killed, the other one thinks they’re okay, it’s polarized. It’s nonsensical. It’s just fascism enabling.”

These sentiments are highly influential on the left, where articles have proliferated with titles like “The Problem Isn’t ‘Polarization’ – It’s Right-Wing Radicalization”, and “Our Problem isn’t Polarization. It’s Fascism.”

The Polarisation Industry

Such critics are responding to a large body of recent scholarship and commentary that links many of the world’s political problems to the extent of division, or “polarisation,” within and between societies.

Much of this discourse focuses on the USA, where Republicans and Democrats famously dislike each other much more than they did a few decades ago. For instance, between 2014 and 2024, the share of Democrats who reported that they would be unhappy if a family member married a Republican rose from 19% to 39%. For Republicans asked about Democrats, it increased from 22% to 33%. Of course, one also finds intense polarisation in many other contexts, ranging from Northern Ireland to Lebanon, the Israel-Palestine conflict to the left/right divide that structures democratic politics in many countries.

Many people view polarisation—especially “affective” polarisation, the fear and dislike of opposing groups—as a powerful force that threatens democracy, social trust, cooperation, and fact-based political debate and public opinion. In highly polarised societies, groups become less willing to compromise and transfer power, more willing to endorse political violence, and more likely to succumb to “tribal” or “sectarian” biases that distort perceptions of reality.

Terms like “tribalism” and “sectarianism” here underscore something important: the attitudes and emotions in highly polarised contexts like the US are not specific to those contexts. They emerge whenever intense intergroup conflict maps onto social identities such as partisanship, ideology, religion, sect, ethnicity, region, or tribe. That is, while every group rationalises their fear and hatred of the outgroup by pointing to its specific crimes, the emotions are strikingly similar and symmetrical across radically different conflicts, whether between Protestants and Catholics, Hutus and Tutsis, or Sunnis and Shias.

This pattern is typically explained in terms of our evolved “tribal” or “coalitional” nature. Our species was forged under selection pressures that favoured powerful motives and abilities for forming alliances designed to outcompete other alliances for prestige, dominance, and resources. So, when we support and identify with a group, automatic “coalitional instincts” are activated. We divide the world into ingroup and outgroup, us and them. We frame group relations as zero-sum conflict for power and esteem. We become obsessed with sending and monitoring signals of group loyalty. And we become instinctive apparatchiks and propagandists, embracing narratives crafted to make our side and its defining narratives look good, and the other side look bad, if not outright demonic.

Polarisation exacerbates those instincts, which in turn exacerbate polarisation, fuelling a runaway process in which competing tribes lose access to a shared reality and a willingness to empathise and compromise with each other.

The Critique

For Stanley and many others on the left, this diagnosis of modern politics is preposterous. Their central objection is that ‘polarisation’ implies a false symmetry, depicting two poles drifting away from a virtuous centre. This means that it can’t capture what the critics take to be a self-evident fact: that the real threat to liberal democracy and social justice comes from the right, the political home of extremism, racism, sexism, transphobia, lies, conspiracy theorising, and—at least in the views of figures like Stanley—fascism.

The problem isn’t that people are divided and tribal. The problem is that one tribe is a sinister menace to society. Treating that menace as an existential, fascistic threat doesn’t involve an irrational “tribal” psychology, an unfortunate hangover from our primitive, evolutionary past. It means seeing the far right for what it is. When confronted with this reality, the appropriate response is not depolarisation (i.e., political moderation); it is to be even more opposed—and hence more polarised—against it.

By misrepresenting this state of affairs and misdirecting our political energy, those talking about the dangers of polarisation and tribalism are complicit in normalising and enabling the right’s attacks on democracy and vulnerable minorities. As Noah Berlatsky puts it,

“A social science that sees polarization and partisanship as the main threats to democracy is a social science that implicitly—and often more than implicitly—is calling for white, Christofascist solidarity against Black (and feminist, and queer, and disabled) demands for justice.”

Am I A Fascist Enabler?

Read more