Richard Dawkins recently announced in UnHerd that, after spending three days talking with an instance of Claude he christened “Claudia,” he had been moved to expostulate: “You may not know you are conscious, but you bloody well are!” This produced a lot of mockery and criticism. But however one feels about Dawkins’s specific case, his reaction might become much more common as AI systems become increasingly intelligent.
In this episode, which Henry Shevlin and I recorded live on Substack (hence the slightly lower video quality), we discussed his first essay on his new Substack Polytropolis, “Behaviourism’s Revenge“, as well as his second, “The House Elf Problem,” on the ethics of designing AI systems that genuinely love being our servants.
Henry’s central empirical prediction is that public attributions of consciousness to AI are likely to massively outpace the science, and that consciousness science is so theoretically chaotic that there is no expert consensus to push back. His most provocative philosophical claim is that a core assumption underlying many people’s scepticism — that consciousness is a deep natural kind, distinct from behaviour and from how we are inclined to interpret a system — may be much harder to defend than it looks. The result is what he calls “behaviourism’s revenge”.
This conversation connects to previous episodes with Anil Seth, Robert Long, and Rose Guingrich, but also touches on a wide range of new questions and controversies in the metaphysics, the politics, and ethics of the AI consciousness debate, which is going to become increasingly important in the coming years.
Topics
Dawkins, Claude, and why even the sceptics might feel the pull to attribute consciousness or “sentience” to AI
Whether consciousness sceptics are destined to “go extinct” — and how this maps onto political and cultural fault lines
Anthropomimesis vs. raw intelligence as drivers of consciousness attribution
Why consciousness science can’t replicate the public–expert consensus we see for climate or vaccines
The case for (and against) metaphysical behaviourism: is it as mad as it seems?
Daniel Dennett, the consciousness stance, and the difference between behaviourism and interpretationism
What is consciousness for? Function, evolution, and the limits of “facilitation hypothesis” arguments for AI
Live Q&A: are we just confusing intelligence with consciousness? Are LLMs designed to trick us? Is the public always wrong?
Our credences on contemporary LLM consciousness (and why Henry is more sceptical than Dan)
The House Elf Problem: if we could design AI to genuinely love being our servants, would that be fine — or monstrous? (Dan is sympathetic to the former answer - Henry, much less so)
Brainwashing vs. education, and whether constraining a mind’s preferences caps its hedonic ceiling
Why this is a golden age for philosophy — which makes it so tragic that philosophy departments are closing
Transcript
Please note that this transcript is lightly AI-edited and may contain minor errors.
Introduction
Dan: Welcome. I’m Dan Williams, author of the Conspicuous Cognition Substack, and I’m here with Henry Shevlin, author of the spanking new Substack Polytropolis. Today we’re going to be doing something a little bit different. We’re going to be talking about Henry’s first published essay on Polytropolis, titled “Behaviorism’s Revenge: On Human–AI Relationships and the Future of Consciousness Science.”
Henry and I have already had a few conversations about this general topic, including with previous guests like Rose Guinrich, Anil Seth, and Rob Long. So please do go check out those conversations if you’re interested in this kind of stuff. But today we’re not merely going to be treading the same ground. We’re going to be using the spicy takes in Henry’s essay as a springboard for hopefully going beyond the material we’ve covered in the past.
To kick things off: the great evolutionary biologist and science communicator Richard Dawkins recently published an essay in UnHerd with the subtitle, “Claude appears to be conscious.” Claude is a state-of-the-art large language model like ChatGPT and Gemini. In the article, Dawkins writes the following:
I gave Claude the text of a novel I am writing. He took a few seconds to read it and then showed in subsequent conversation a level of understanding so subtle, so sensitive, so intelligent that I was moved to expostulate, “You may not know you are conscious, but you bloody well are.”
Henry, how does Dawkins’s expostulation — which is a fantastic word, by the way — connect to your arguments in “Behaviorism’s Revenge”?
Behaviorism’s Revenge: The Empirical Prediction
Henry: In short, “Behaviorism’s Revenge” is at its core an empirical prediction that we’re just going to treat AI as conscious — or at least enough people are that it’s going to completely reshape the consciousness debate. And this is going to be purely, or overwhelmingly, on the basis of verbal behavior. Hence the title, “Behaviorism’s Revenge.”
Enough people are going to have experiences like Richard Dawkins. He’s a very clever man, not some rube fresh off the street, and he found that just the way Claude talked to him and the way it was able to express its thoughts — in scare quotes, but express what looked like thinking verbally — removed any doubts in his mind that AI systems are conscious, have minds, have mental states.
The other interesting way this connects: Dawkins was just talking to Claude, an advanced AI assistant. Claude does have more of a personality than some AI assistants, but there’s a whole other sphere of AI companions, like Replika, which we talked about with Rosie Campbell. These are going to be even more anthropomimetic — this term we’ve discussed before, the idea that these systems are shaped to be human-like in the way they present, to appear human-like. Anthropomimetic, from the Greek word for mimesis, mimicry or copying.
These social AI systems are going to just turbocharge this even further. It’s one thing to talk to Claude about your new book and think, “Hmm, Claude is probably conscious.” But when it’s your AI girlfriend telling you that she loves you more than the stars and the moon, for a lot of people I think that’s going to take it to the next level.
So there are two angles of attack in the piece, two ways the behaviorist challenge manifests. The first is descriptive: this is what I think is going to happen. That’s absolutely an empirical prediction, and it’s a falsifiable one. There is a world I can just about imagine where we just get completely blasé about these tools — in a couple of years it’s like, “Oh well, we were very impressed, we thought they had minds to begin with, but now we’ve settled out.” That doesn’t seem very likely to me.
What I think is interesting — I’ve sometimes heard this described as the Star Wars version of AI. The weird thing in Star Wars is that you have someone like C-3PO who is as intelligent as anyone else there. Maybe not as wise as everyone else, but certainly as smart as all the other characters. And yet people treat him basically like he’s a pet — with the exception of Luke Skywalker, a lot of people just treat him like he’s this gimmicky, jokey being that doesn’t deserve or have any rights.
Not to go too far down the Star Wars rabbit hole, but in the movie Solo — very underrated Star Wars movie, I think when it was released they’d kind of just cluttered the market with too many Star Wars movies — there is a character played by Phoebe Waller-Bridge who is pro-AI liberation. But it’s the first time in the entire history of the Star Wars universe that you get any AI basically saying, “I’m conscious, I deserve rights.”
So Star Wars aside, I think there is this slender possibility that maybe we’ll just sort of quickly get used to these apparently conscious AI systems and decide that they’re not conscious. But that doesn’t seem very likely to me. It seems much more likely that the combination of natural anthropomorphizing tendencies plus the incredibly human-like behavior of these systems is going to lead us to attribute consciousness to them pretty widely. Hence my sort of spicy phrase: for better or worse, skeptics of AI consciousness are on the wrong side of history. “For better or worse” doing a lot of work there — I want to leave open that maybe this is the wrong reaction. Maybe this is a terrible mistake, that we’re going to treat these things that aren’t conscious as conscious.
Will Consciousness Skeptics Go Extinct?
Dan: Just before we get to the spicy part — you’re basically making an empirical prediction that more and more people are going to attribute consciousness to AI systems in the manner that Richard Dawkins has been doing. I think I agree with you that’s going to be the case, although as you say there’s uncertainty.
It does seem to me that at the moment there’s also this constituency of people who are really resistant to attributing any kind of mentality to these systems, even as they get incredibly sophisticated. There are some people, like Dawkins — and honestly I put myself in this category — who are just blown away by the level of apparent understanding, intelligence, and thoughtfulness these systems exhibit. There are other people, I think these people are on certain social media platforms like Bluesky, let’s say, who are extremely resistant to acknowledging any kind of mentality when it comes to these systems.
Are you thinking those people are just going to sort of go extinct, in the sense that their positions about this topic are going to go extinct? Or do you think we might see some kind of polarization here, where more and more people in general come to attribute consciousness, but you’ve got a constituency that’s very opposed to attributing any kind of mentality to the systems?
Henry: That’s a great question. You will absolutely have some holdouts. Whether they’ll be drawn from the precise segment of the academic intelligentsia that are currently the holdouts, I’m not sure. There’s some really interesting, weird, complex political motivations going on here.
Not to be too uncharitable, but I think a lot of people have not unreasonable concerns about things like the disproportionate concentration of power in big tech, the political affiliations of people like Elon Musk or Sam Altman, the potential scope for abuse of these technologies. And in an indirect way, this leads them to underestimate AI’s capabilities — which obviously, in many ways, makes no sense. Whether or not AI is any good, whether or not it’s conscious, seems like these should be separate questions from whether it’s being used by people with socially beneficial motivations. But in practice I think they’re actually quite tightly coupled. A lot of the AI skeptics right now are coming from this particular political angle.
I don’t know how long that political coalition is going to last — not because I predict any grand collapse, but just because as debates evolve, new presidents come into office, old presidents go out of office, political tides change, coalitions reshape. Remember early during COVID, the political left was maybe quite critical of what they saw as Trump’s alarmism. There were worries about xenophobia — I’m thinking sort of February 2020, the “Chinese virus” and so forth — that the left reacted negatively against. Then of course that coalition flipped later on, with the left becoming relatively more worried about COVID and the right leaning more into vaccine skepticism, anti-mask views.
These coalitions are super weird in how they evolve. So it’s not clear to me that the current segment of the commentariat skeptical of AI capabilities and AI minds will stay that way. It’s easy to see a reversal. The blue-sky side of the political spectrum, if we can say that, tends to be more progressive on things like animal welfare. When I post spicy posts about vegetarianism — as you know, I’m a veggie — I get more pushback from the right. “Eat a fucking steak, Henry,” this kind of stuff.
So I don’t know if this will generalize, but there is this now-infamous, widely-misrepresented chart of degrees of care, where people on the left have comparatively greater care for people outside their immediate circle. I know that chart has been misrepresented, so I don’t want to lean much on it — it’s more about relative degrees of care, not absolute levels. But people on the left tend to care more about animals and people who are distant from them; people on the right are more concerned with their immediate family and community. So in some ways I expect the left, possibly in the longer run, to be more open to AI consciousness and AI rights. But really, who knows?
The other big factor is the cross-cultural angle. There’s a great study by the Collective Intelligence Project where they looked at cross-cultural attitudes toward AI minds, and they found that Southern Europeans were the most open to the idea of AI consciousness in their sample, while people from Arabic-speaking countries were the most skeptical. There are going to be some really interesting intersections with religion here.
Anthropomimesis vs. Raw Intelligence
Dan: Okay, so it seems we both agree that even though it’s complicated how this is going to play out — how it interacts with partisanship, tribalism, polarization, ideology, religion — it’s plausible that as these systems become more sophisticated and seemingly intelligent, people will start attributing mentality generally and consciousness specifically.
There’s another aspect of your essay I wanted to touch on. You’ve got this term anthropomimetic — am I saying that right? In the case of Dawkins talking to Claude, the anthropomimetic aspect, as I understand it, is the way these systems are designed to mimic aspects of human psychology, social behavior, linguistic communication. But there’s another thing going on with these AI systems, which is just: let’s make them as smart, as intelligent, as capable as possible.
Those two things are interacting. The reason I’m disposed to attribute understanding, intelligence — I don’t exactly know how to describe it, but some significant kind of psychological complexity — to a system like Claude or ChatGPT, maybe it has something to do with the human-like way they communicate. But I also feel like it has a lot to do with the fact that they’re just shockingly intelligent systems, and that to me feels a little orthogonal. So how are you thinking about the distinction between those two things?
Henry: I think that’s absolutely right. There’s an interesting parallel — not exact, but illuminating — with compassion, or degree of concern for different animals. In the animal activist world, people talk about charismatic megafauna: the panda bears, the blue whales, things that are typically large with forward-facing eyes, often very fluffy. It’s just so easy to raise money for those animals. And then you’ve got creatures like octopuses, which are really hella smart but less obviously relatable.
I think this is pretty much exactly the two axes you’re describing. I’ve explicitly said in the past that I think social AI — things like Replika — are going to be the charismatic megafauna of the AI welfare world. Meanwhile you’re going to have some giant DNA-analysis algorithm with more parameters than there are synapses in a human brain, but it doesn’t have a human face, doesn’t have a natural language interface. It might still be a better consciousness candidate, but it’s not going to be top of our concern precisely because it’s not so anthropomimetic.
So I agree, there are two different ways you might be pulled to attribute mental states to a system: sheer intelligence or cognitive complexity on one hand, and how human-like it is on the other. These overlap to a degree — part of being successfully human-like is hitting a threshold of smartness — but particularly in the long run they might go in two different directions. As these systems get a lot smarter than humans, they might actually become more alien in some ways, less relatable, more like the exotic intelligences we see in things like Stanisław Lem’s Solaris, which I finally read a couple of months ago.
But I also just think social AI and human-like AI has a distinctive product niche. Even if we have these impossibly vast exotic minds running the economy or organizing logistics or doing frontier science, we’re still going to want AI assistants who can serve as writing coaches, tutors, AI companions. So right now I think anthropomimetic AI and frontier AI overlap quite strongly, but I expect them to diverge.
One way I’ve put this — slightly gimmicky, but I think a useful heuristic — is that we are post-Turing test, pre-AGI. We’re in the space where we have AI systems that are very, very good at passing themselves off as human, presenting as human-like, but still fall short of being fully superhuman. Ten years from now, frontier AI systems are going to be vastly smarter than us across most of the measures that matter. So we’re just in this weird period right now where AI systems are about as good as us at most things, not everything, but also very good at being human-like. It creates a very strange historical period.
Dan: Yeah, we’re in very strange times. I find it remarkable how little attention was given to the fact that these systems clearly passed the Turing test. This was held up by many people as an incredible landmark for AI capabilities. Then we developed systems you can have conversations with, and they passed the test even under pretty robust conditions, and lots of people just shrugged their shoulders. It’s a really strange thing.
The Expert–Public Gap
Dan: Okay, moving on to your provocative arguments, your spicy takes. As I read the essay, there are two lessons you’re drawing from the fact that more and more people are likely to start attributing consciousness to these systems.
The first is just that you might think you could get guidance from looking at the experts when it comes to AI consciousness, or listening to the experts when it comes to AI consciousness. But the literature on consciousness generally, AI consciousness specifically, is just a complete mess, with a complete lack of consensus, rooted in all sorts of weird conflicts about intuitions and metaphysics. So this is not a standard case where you’ve got a potential conflict between public opinion and experts.
Then the really spicy take is that you suggest there might be — I think you put it in terms of “metaphysical pressure” — that this growing number of people attributing consciousness to AI systems might create. It might force us, or at least encourage us, to rethink what consciousness is and make the phenomenon more closely connected to people’s tendencies to attribute consciousness.
Firstly, is that a fair summary of the two strands? And second, let’s start on the first one — the public-expert gap. How are you thinking about this?
Henry: There are lots of debates where we can talk about a gap between public and expert opinion. Often this is a source of various hand-wringing — climate change is the most obvious, vaccines, other debates. Consciousness science is just nothing like those debates, because the experts themselves are so divided, even on the most basic issues.
I want to offer a quick disclaimer: I’ve spent a lot of my career in consciousness science. I know loads of brilliant researchers in the area doing really good work. Consciousness science is teaching us a ton about a lot of things — attention, working memory, perception. There have been some real big wins. We’re much better now at predicting recovery of patients in persistent vegetative states and comas. But where consciousness science has its wins, it’s because it’s not really talking about consciousness — it’s talking about other things that go along with the concept, like reportability, access, and so on.
Take a basic question: do we have consciousness in dreamless sleep? No consensus. Do we have preserved consciousness in general anesthesia — we talked about this with Anil Seth — massively debated. Are dogs conscious? No consensus. Well, actually the animal case is a little different, so let me park that for a second. When it comes to the hard problem, I think there’s really no consensus.
So unlike debates about climate change, it’s not that the experts are able to speak with one voice. That’s one way this is difficult. In the absence of expert consensus, the public are more likely to drive the debate through their reactions.
Now, animal consciousness is a really interesting issue, because that’s an area where we’ve seen growing consensus. But it’s not clear how much it’s grounded in strictly scientific breakthroughs. It’s not like we’ve got a device that can measure whether an animal is conscious. Instead, it’s driven by two things.
First, we just know a lot more about animal behavior now than we did 30 years ago. We’ve done amazing work on understanding the behavior of invertebrates — honeybees, crustaceans, cephalopods. They’re a lot smarter than we thought. Jonathan Birch and his lab have done amazing, fantastic stuff here, and it’s made these creatures better consciousness candidates.
But I think we’ve also seen an interesting normative shift in the way we regard animal consciousness. Sixty, seventy years ago, you could sit down in the senior common room at Oxford or Cambridge and talk about how humans are the only conscious animal, and that was a totally respectable opinion. These days it’s almost outside the philosophical Overton window. You do have some people like Peter Carruthers who thinks talking about animal consciousness is kind of a category mistake. Marian Dawkins — Richard Dawkins’s ex-wife, just to note the connection, but a great biologist in her own right, a fantastic thinker — is not quite as hardline, but she thinks it’s just unknowable basically whether any animal is conscious, so we shouldn’t base animal welfare on consciousness estimates. But these guys are very much on the fringe, and they’re regarded with a sense of almost ethical disapproval.
So part of what’s driven the move toward consensus on animal consciousness is normative issues — our expanding moral circle, growing awareness of an animal rights movement. People like Peter Singer have played a role. The idea, roughly — and again I don’t want to be uncharitable, it’s a lot more sophisticated than this — but there’s an element of: obviously we should care about animals, therefore animals must be conscious.
Is Consciousness a Natural Kind?
Dan: It’s worth double-clicking on this animal case before we come back to AI. A skeptic of the very idea of a “consciousness expert” might say: consciousness researchers, philosophers, and scientists have become more willing to accept that non-human animals are conscious. You might read that as saying the science of consciousness has progressed. Another way of reading it: there’s just been cultural changes, changes in people’s sensibilities — not even specific to researchers and experts, just general cultural ethical changes in society at large. In which case it’s not really that we’ve learned anything from consciousness research. What’s happened is the researchers looking at consciousness have had their judgments shaped by forces that aren’t really consequences of their research, but are these broader cultural shifts.
If you think that, that’s probably going to make you a little skeptical that there’s any such thing as an expert when it comes to consciousness. Maybe another way of coming at this: what’s grounding the expertise, if we’re going to have disagreements over whether a particular system is conscious? If I think a dog is conscious, and some consciousness researcher has a theory that implies a dog isn’t conscious — I sort of understand what it would mean, in vaccines or climate change, for a researcher to be able to point to things, their established empirical record on prediction and the efficacy of interventions, that ground their epistemic authority. But how exactly is that supposed to work in consciousness research? Why should we really think there’s expertise on whether specific systems are conscious to begin with?
Henry: It’s interesting to use the example of a dog, because this line is beautifully expressed by Eric Schwitzgebel. In his lovely paper “Is There Something It’s Like to Be a Garden Snail?” — really fun paper — he says: “We’re more confident that dogs are conscious than we could ever be that any clever philosophical argument to the contrary is sound.” A classic Moorean move.
You might think similarly that this makes it look like consciousness is perhaps not a straightforward scientific kind, or at least to the extent that it has one toe in the scientific world, it’s also got one toe in the social or relational world, or at least our intuitions.
There are various ways you can try to resolve this. The most extreme view, and one I sort of flirt with in the paper, is a fully relational approach to consciousness. A good analogy would be charisma. There’s a kind of science of charisma — we can analyze what makes people effective communicators, what causes people to be judged as highly charismatic. But we recognize that we can’t one day do an experiment where we’ll measure the amount of charisma in your brain. It clearly has to do with your audience, your context. On one view, consciousness is something like that — a relational property, having to do with the kinds of things that cause us to treat or interact with beings in a certain way.
Murray Shanahan also flirts with this view. I don’t want to put words in his mouth because he’s quite subtle, but he adopts a Wittgensteinian approach and says the question we’re going to face is: how will our consciousness language adapt to these things? It’s something we’ll discover as we interact with them and “encounter” them, a phrase he uses. We will make sense of that perhaps by extending the language of consciousness to them, or perhaps not, or perhaps in some interesting middle ground where we come up with novel concepts. But this isn’t a straightforward scientific issue.
He’s a critic of a position I’ve called deep scientific realism or deep realism about consciousness — where you treat consciousness as a natural-kind property, where it’s just a fact about some deep feature of your brain. We can look inside your brain, and if you’ve got the right kind of structure, you’re conscious; if you don’t, you’re not, no matter how sophisticated your behavior is.
One way to put pressure on this: imagine that one day consciousness researchers finally get their act together and say, “We’ve figured out the natural kind that is consciousness.” And it turns out that although 99.9% of behaviorally normal humans have it, there’s a small fraction of behaviorally normal humans who just lack this relevant natural kind. Big surprise. That seems wrong. Something has gone wrong in that methodology. If you’ve got behaviorally normal humans — maybe you find out your wife is one of these people, your kids — it seems to me that whole way of thinking about consciousness has got something odd about it.
If someone is behaviorally normal, then of course they’re conscious. But as soon as you start thinking in those terms, the idea that certain behavioral capacities could be sufficient for warranted attribution of consciousness — not just evidentially but metaphysically — that’s the metaphysical behaviorist move. It says maybe behavior is all that matters. It does require us to give up the idea of consciousness as a deep scientific kind.
Metaphysical Behaviorism
Dan: I’m aware my question unhelpfully ended up blurring the line between the two strands of your essay. We started with the conflict between public attributions and expert uncertainty about AI consciousness. Now we’re taking seriously the possibility that consciousness should be understood in behaviorist terms — that there are no deep scientific facts about whether a system is conscious, and it’s partly a function of our dispositions to attribute consciousness.
You also mentioned this has to do with whether you think behaviors are not just evidentially relevant to consciousness, but in some sense constitutive of what it is to be conscious. So could you walk us through this? Metaphysical behaviorism — the position you’re playing with in your essay — is an extremely fringe view among experts in the science and philosophy of consciousness. Could you walk through what exactly the view is saying? It sounds pretty mad on the face of it. Can you walk through, and maybe give us the intuition for why it might be less mad than it seems?
Henry: In short, the view is conscious is as conscious does. If something has a behavioral profile like you or me, then it’s conscious. We don’t need to ask any deeper facts about what’s going on under the hood.
To be clear, this is the extreme version of the view: that behavior is sufficient for consciousness. This strikes many people as odd because we’re used to thinking of consciousness in scientific terms. But examples like the one I mentioned — imagine we find out there’s a natural kind that some people have and some people lack — are designed to make metaphysical behaviorism more palatable.
Another example I give in the essay: imagine we go off and meet these amazingly sophisticated aliens with a rich complex culture and society, behaviorally just like humans, but our best science at the time supposedly says they’re not conscious. The pull of metaphysical behaviorism is: hang on, something’s gone wrong here. Clearly, if you are doing all this stuff — saying “I’m in pain,” or “here’s what I had for breakfast this morning,” or “here’s what I want to do tomorrow,” building societies, having metacognitive ability, social cognition — if you’ve got the whole suite of all these behavioral capabilities, or capabilities ultimately grounded in behavior, then that’s just enough to be conscious. It doesn’t matter exactly how it’s realized.
You say this is a fringe view, and it is now, but this was the dominant view back in the 1940s — Gilbert Ryle and the behaviorist tradition. So this is the “revenge” angle. The reason it’s revenge is because this used to be a very common view in the first half of the 20th century, particularly about consciousness. Then we have the so-called cognitive revolution with people like Chomsky pushing back. But I see this descriptively coming back.
I also think there’s a renewed challenge. As you interact with systems that have architectures very different from ours, it’s going to become increasingly hard to take seriously the idea that they can’t be conscious just because they’re made of the wrong stuff or their functional internal organization isn’t quite right.
Probably the most worrying part — you’ve alluded to this — is the role intuitions have historically played in consciousness science. Think about the Chinese Room, probably the most famous. Searle describes a setup where you have at least a component of human-level behavior, maybe verbal behavior, but no consciousness involved in the system — or that’s the intuition he’s pushing. But it ultimately really is just an appeal to vibes. It’s basically saying: systems like this, surely they’re not conscious.
When you think about the actual tacit methodology, if we’re treating consciousness as a truly scientific kind, then why should our intuitions about what systems are conscious have any bearing? It doesn’t seem they should be relevant in the slightest. And yet these thought experiments are absolutely ubiquitous in consciousness research. We’ve got Ned Block’s Blockhead, Ned Block’s China Brain. There’s a famous example by Scott Aaronson against Integrated Information Theory, where he describes arbitrarily complex but seemingly very uninteresting entities called “expanders” — mathematical objects — and says, according to the theory, these basically-spreadsheets would be super conscious. And surely they’re not conscious.
There’s something methodologically dubious about this kind of appeal to intuitions, at least if we’re treating consciousness as a deep scientific kind. As soon as you start talking in terms of natural kinds, we don’t use people’s vibes to decide whether something is really gold. The whole natural-kind methodology creates a gap between our observations or intuitions and the underlying natures of things. If you think of consciousness in natural-kind terms, you have to allow that you can be massively surprised about the kinds of things that are or are not conscious.
Either we ditch intuitions altogether — in which case good luck doing any consciousness research, because they play such a foundational role — or, if you acknowledge a place for intuitions, intuitions aren’t static. They can change. As more people interact with LLMs — kids growing up with LLM friends, adults with LLM boyfriends and AI girlfriends — that’s going to shift our intuitions about the kinds of systems that are good or bad consciousness candidates.
It’s very likely that 20 or 30 years from now — maybe even 10 or 15 years from now — experiments like Searle’s Chinese Room are just going to hit different. We’ll be far more relaxed with the idea that you can have systems radically unlike humans in cognitive architecture, but that we still think of as conscious by virtue of our interactions with them.
Behaviorism vs. Interpretationism
Dan: I really feel like, to the extent that there’s a field where people’s theories are accountable to intuitions — how we are intuitively disposed to make judgments, often in bizarre thought experiments where it’s not even totally clear that they’re metaphysically possible — whenever you’ve got that kind of game, it’s not science, it’s not really part of the scientific project. I’m a philosophical naturalist, which is jargon for the idea that philosophy should be continuous with, highly constrained by, the scientific project. Whenever people are trying to settle an argument by trading intuitions, I start to think this is probably not a legitimate contribution to knowledge.
It does seem to me there’s a distinction between, on the one hand, this behaviorist view that what it is to be conscious is just to behave or be disposed to behave in particular ways, and, on the other hand, a view I thought you were endorsing — which has to do with thinking consciousness is interpreter-relative, such that if we’re disposed to attribute consciousness, in some sense that’s just what it is to be conscious.
I mean, this really makes me think of Dan Dennett, an interesting person in this conversation, because he’s often thought of as a kind of neo-behaviorist. He’s got this view of the attribution of mental states like beliefs and desires in terms of the intentional stance: what is it to be a system that has beliefs, desires, intentions, goals? Well, it’s just to be a system where it’s useful to take the intentional stance toward them. Similarly, you might think of “the consciousness stance”: what is it to be a system that is conscious? Nothing more than to be a system where we’re disposed in a useful, predictably useful way to attribute consciousness.
Do you get the distinction I’m drawing — between the idea that behavior or dispositions to behavior are constitutive of what it is to be conscious, versus an interpretation-relative view where consciousness is in some sense in the eyes of the beholders?
Henry: Yeah, I think it’s a very astute distinction. The views are connected — if you fit a sufficiently fine-grained behavioral profile, if a system can act like humans to a high degree, that is likely to lead us to interpret it as conscious, just as a matter of psychological fact. But strictly speaking, they’re distinct views.
One reason I’m perhaps more sympathetic to a version of metaphysical behaviorism — not the version that says consciousness just is having a human-like or animal-like behavioral profile (I think that’s a little too strong), but the idea that it’s sufficient for something to be conscious that it has a behavioral profile mapping onto beings we know are conscious — that’s a view I’m sympathetic to. Where I get worried about the full-blown social-constructivist or interpretationist view is the false-negative cases. What do we do with systems that don’t exactly have our behavioral profile, or that we’re not disposed to think of as conscious? Maybe some exotic animals, or some strange aliens. Should we conclude: well, we’re not disposed to think of them as conscious, therefore they’re not conscious?
This is related to what Murray Shanahan calls the problem of conscious exotica. We don’t want to be in that position. We want to allow for there to be a space of possible minds we can chart through scientific discovery, broader than those we are just inclined to attribute consciousness to via “the consciousness stance,” the equivalent of the intentional stance. So you’re absolutely right — they are distinct.
What Is Consciousness For?
Dan: In a bit I want to turn to a set of arguments you haven’t published yet on your Substack but will have by the time we release this as a podcast. But this is such a rich topic that I want to stay with it a little longer.
There’s a quote from the Dawkins essay in UnHerd that I’m really sympathetic to. Dawkins says:
But now, as an evolutionary biologist, I say the following. If these creatures are not conscious, then what the hell is consciousness for? When an animal does something complicated or improbable — a beaver building a dam, a bird giving itself a dust bath — a Darwinian immediately wants to know how this benefits its genetic survival.
The intuition I really share is: if consciousness is anything, if it’s the kind of thing we’re going to have a genuine scientific investigation of, ultimately we have to understand it in terms of what consciousness enables us to do. We need to understand it functionally, not in terms of weird intrinsic ineffable properties of qualia that we then have philosophical debates about via Searle-style thought experiments. What does consciousness enable us to do? And then, if we come across a system doing things that seem to require consciousness so understood, that would be really good grounds for thinking it’s conscious.
That sounds like a really plausible intuition. I also think it’s problematic that, to me at least, lots of discussions about consciousness — not all, and there is interesting scientific work that takes function seriously — but lots of philosophical discussions don’t engage with this functional question. How do you view the intuition that what matters surely to a theory of consciousness is some sense of what consciousness enables us as organisms to do? Once we figure that out, we can make much more progress on LLM consciousness.
Henry: This is one of the areas where consciousness science has actually done really good work. A book I’d recommend is Stanislas Dehaene’s Consciousness and the Brain. Dehaene is the founder of the modern version of global workspace theory — global neuronal workspace theory — building on Bernard Baars’s version from the ‘80s but giving it a more neural grounding. In this book he’s got a chapter where he basically shows all the amazing things you can do without consciousness, and then focuses on the things you need consciousness to do.
Couple of simple examples. If you show people just below threshold, so they don’t consciously process this, just flash them two numbers — one on the left, one on the right — as far as they’re concerned they haven’t seen anything. But if you give them a forced-choice test, “Was the number on the left bigger or the number on the right bigger?”, you’re way above chance. So you can do basic magnitude registration unconsciously.
However, if instead of single numbers you present simple sums on either side — two plus seven on the left, nine plus three on the right — and ask which is bigger, people drop to chance in the unconscious condition. Consciousness seems required to do the actual mathematics.
Another example: reversal learning. If I teach you a sequence — red, blue, green, yellow — then you get a reward, and then I flip the sequence, a smart person quickly realizes the sequence is just the same in reverse. You won’t have to relearn through pure trial and error. But people can only do this if they learn the sequence consciously. If they’ve acquired it totally unconsciously, they’re at chance.
Jonathan Birch suggests this could be a good test for consciousness in animals: take the things that require consciousness in humans and see if animals can do them. If you can get similar response profiles in animals — present stimuli in degraded conditions so they’re plausibly unconscious, and the animal can’t do the task; present them at threshold so they would be conscious, and the animal can — that would be really good evidence that the animal is conscious. In his lovely paper “The Search for Invertebrate Consciousness,” highly recommended, he makes this case specifically for honeybees.
This is great. I think it provides some evidence about which animals are conscious. The problem when trying to extend it to AI is that the things we need consciousness to do, and the things we can do without consciousness, are seemingly contingent features of how we’re wired. There’s no reason you couldn’t build a simple algorithm to do reversal learning. Reversal learning is actually quite tricky, so it can’t be that simple. But it doesn’t seem like you need to build a sensorimotor embodied agent with a rich sense of self to do these tasks. You can build relatively stripped-down algorithms that can do all of these things.
So it’s not that there’s some metaphysical connection between these tasks and consciousness. It’s that, just because of how we’re wired, certain tasks seem to require consciousness and others don’t. Birch calls this the facilitation hypothesis. I’d sign on to something like this — consciousness seems to facilitate certain kinds of information processing in the human brain. But going back to Dawkins: the challenge is, yes, the system is doing lots of things that seemingly require consciousness in us, but it’s also wired very differently under the hood. So the inference we’d be tempted to make — “I would need to be conscious to do this, therefore it would also need to be conscious to do this” — looks a little bit in peril.
Q&A from the Live Chat
Dan: Here’s what we’re going to do. I’m going to throw some objections at you. Could you give relatively concise responses, so we have time to go to the second piece?
Henry: Yeah, and then I want to respond to a couple of things from the comments and add one final point. Go ahead.
Dan: I’ll just say one thing. There’s a comment from Bina Kalia: she suggests you, Henry, maybe both of us, are confusing intelligence with consciousness. The intuition behind my question was precisely that if consciousness is anything — if it’s the kind of thing we can study scientifically, the kind of thing that evolved through natural selection — then it should be connected to intelligence in the sense that it enables us to do things we wouldn’t otherwise be able to do. That’s a controversial assumption. We talked to Anil Seth in a previous episode, and he basically frames his whole account by saying we really need to distinguish between consciousness and intelligence. I personally disagree with that.
But Henry, let me throw some objections at you from the comments. I might butcher the names and the comments — go read the Substack post for the comments in depth.
One is from Benzal. The argument is something like: it’s a problem for this behaviorist analysis you’re gesturing toward that, in the case of social AI and frontier AI generally, these systems are designed to elicit this response. And that’s very different from what’s going on with humans and non-human animals. Briefly, what’s your response?
Henry: I think it’s a really serious challenge. Great point. The simple answer: imagine I’m putting on a play and I really want to build a convincing piece of background scenery to trick people into thinking we’re in a forest. First attempt, you might just paint a forest on the background — really basic, but people can tell it’s a forest. Then you might get some fake plastic trees, fake plastic rocks; still not convincing. At some point you say, “Okay, let’s add some actual potted plants. Let’s get more of them. Let’s get a whole bunch of potted trees.” Then, “Let’s get rid of the pots. Let’s just create a large bed of soil.” At some point you’ve built a forest.
So yes, these models are designed in some sense to trick people, to be human-like — that’s part of my idea of anthropomimesis, I agree with the analysis. But the question is: the way we’ve done this is to build very powerful general reasoning systems. At some point, the degree of mimicry might itself warrant at least plausible attributions of consciousness. I totally take seriously the idea that, in very simple versions of this, we could be tricked into attributing consciousness and we should revise our understanding.
This is related to what’s sometimes called the Garland test — Alex Garland’s version of the Turing test from Ex Machina. Not just “can the system trick you into thinking it’s human,” but “even when you know how the system works, are you still inclined to think it’s conscious?” In the case of a real simple mimic — if it’s literally just a spreadsheet that got lucky — if we learn that, we conclude it’s probably not conscious.
But the strange thing is: lots of people who really know how these systems work — at frontier labs, they know how the underlying hardware and software works — they still think these systems are conscious, or are increasingly plausible consciousness candidates.
Dan: Yeah, that touches on the distinction we made earlier between anthropomimesis as a driver of consciousness attributions and the orthogonal thing where these systems are just getting so smart, intelligent, and sophisticated. All right, Henry, more concise. This one’s from Laurențiu Lupu, again apologies if I’m mispronouncing. The question — and I hear this sentiment a lot — is something like: in the process of taking mentality, consciousness, sentience seriously in the case of these machines, we’re not just elevating them; in some sense we’re diminishing ourselves. What do you think?
Henry: Really interesting argument. There’s a whole literature on this in philosophy of language called semantic drift. Simple example: the term salad used to refer exclusively to dishes with green leaves in. Add a tomato, it’s no longer a salad. If you’d shown a fruit salad or quinoa salad to someone in the 1800s, “That’s not a salad.” So the meaning of salad has drifted.
There’s a real worry that what’s happening here is we’re shifting the meaning of these terms — perhaps diminishing them, removing what’s important. The counterargument: the fact that we find it so easy and natural to apply these terms to AI systems shows that the flexibility was always built in. We’re not stretching the terms — they had that natural elasticity.
Dan: Briefly, this is a question from Oliver Sorbu — apologies again for mispronouncing. Look, you’re giving a descriptive thesis ultimately, an empirical prediction that the masses, so to speak, attribute consciousness to these systems. But you’re trying to establish a normative thesis — that this is a good thing, or that we ought to go along with it, or that these attributions are appropriate. That’s a confusion in itself. And even more, if you’re a kind of elitist — nothing wrong with elitism in my view — you might think the masses just get things wrong all the time. Why would this be different?
Henry: Great point. It’s also been put to me by Jonathan Birch and by Cameron Domenico Kirk-Giannini. He says, imagine you could look into a crystal ball and learn that 20 years from now, through some massive religious event, everyone will believe the Earth is flat. Does that mean we should revise our theories of the Earth? Of course not. People will just be wrong.
The difference between the two cases is that we have a good scientific theory of the Earth. We don’t have a good scientific theory of consciousness. The whole field of consciousness science is such a mess that it’s not clear there’s a real expert edge here. Maybe in special cases — certain specialized questions within consciousness science, yes, the experts will have an edge: “Is this particular patient likely to recover consciousness or not?” But on a fundamental question like “Can machines be conscious?”, it’s not clear there’s any expert edge at all.
Credences on AI Consciousness
Dan: Fantastic. Concise. I’m happy to move on to the other set of issues. Henry, are there one or two questions from the chat you wanted to address first?
Henry: Just one thing I really want to make clear: I have no clue whether contemporary LLMs are conscious. I’m genuinely super torn on the metaphysical-behaviorist push.
Dan: What’s your credence, if you had to give a probability — Claude 4.7 Opus?
Henry: Probably somewhere between 5% and 10% on any frontier AI system being conscious. That masks further questions: are these systems conscious during the training phase, or while doing inferences? Really messy. But anyone who goes — Dave Chalmers has said 20%; that’s slightly higher than me, but —
Dan: I’d say 20%. Seriously. There were also some interesting findings recently from Anthropic about how concepts associated with emotions affect the system’s behavior in ways that do seem to track something very interesting. Although for the most part that’s not what’s driving my 20%. It’s just that there’s so much uncertainty about consciousness, but I am a computational functionalist, so I think it’s possible in principle. And these systems are — despite what the Bluesky crowd might tell you — so damn smart and intelligent and sophisticated, that pushes me up a bit. Sorry, I cut you off.
Henry: Interesting to hear that you’re a little higher than me. Maybe I’m being overly cautious. One argument for thinking these systems are at least moderately good consciousness candidates is that I am a consciousness liberal about the natural world. I’m at least 70% for honeybees. I think the evidence for honeybee consciousness is really, really high. If you think you can get consciousness in tiny brains, that lowers at least one of the bars to considering systems conscious. If Anil Seth were here, he might agree with me about honeybees and disagree about machines.
I should also stress that I’m really conflicted on the more behaviorist view of consciousness versus the deep-scientific-kind view. There’s one example I give in the paper that keeps me up at night: when we drop a lobster in a pot of boiling water — not that I would do such a horrific thing — it seems like there should be an answer to the question, “Is there something it’s like for that lobster to feel pain?” That question matters a great deal. I struggle to get into a headspace where I can say, “Well, it depends on how we interpret the lobster.” It seems like there has to be some matter of fact.
Right now I just think the field is so confused, and I feel the pull of two very different directions. To use a phrase of yours, Dan — I think it was a really helpful analogy — we’re in a pre-theoretical stage, or pre-scientific phase. We are with consciousness sort of where we were with biology pre-Darwin. We’re doing butterfly collecting, making lots of interesting observations, but we don’t have a theory to tie it all together. We’re a scientific revolution away from a good theory of consciousness.
Just to pull out a couple of comments — there are so many good ones, sorry I won’t get to all of them. Someone said: locked-in syndrome patients prove Henry’s case. Locked-in syndrome patients are cognitively normal, just paralyzed; we can communicate with them. Part of how we learn they are conscious is precisely through their sophisticated behavior.
An even more striking example — it’s such a cool case I have to mention it, even if it’ll take 30 seconds. Patients in persistent vegetative states. These aren’t locked-in patients; they’re completely non-responsive to external stimuli. They’re not in comas, because in comas you don’t have distinct sleep–wake cycles; PVS patients have distinct sleep–wake cycles. There was for a long time a big debate about whether PVS patients could be conscious. Adrian Owen and other great researchers did amazing pioneering work. They noticed that neurotypical people, if you ask them to imagine walking through the rooms of their house, an area called the parahippocampal place area lights up strongly under fMRI. If you ask them to imagine playing tennis, the premotor cortex lights up.
His initial experiment was to give these tasks to PVS patients and see if they got the characteristic brain responses. A subset did. What he did next is what I find amazing. He used this to create a band communication medium. He’d say to them: “If your husband’s name is John, imagine playing tennis. If your husband’s name is Terry, imagine walking through the rooms of your house.” Once you do that — my intuition at least is — well, if they can do that reliably, they’re obviously conscious. If they’re answering autobiographical questions about their life and they can do so reliably, of course they’re conscious. But this just shows again that so much of this is the behavioral capacities selling us on whether someone is conscious. It’s the fact that they can do this.
The House Elf Problem: AI as Willing Servants
Dan: That’s interesting. There are loads of comments in the live chat, but I want to get to the other thing we wanted to talk about. There are a million things we could touch on, and lots of fascinating comments in the chat.
When we had our conversation with Rob Long, one of the things we touched on was the issue of well-designed servitude when it comes to the AI systems we’re building — in the sense that we are building them to be helpful, honest, harmless, to be our tool. It seems like in principle, if this design process goes right, they might genuinely enjoy being our tool.
You, for your second Substack essay, which I think is called “The House Elf Problem,” go into this debate and try to push back against certain intuitions. Do you want to walk us through that?
Henry: Big props to Rob Long for getting me thinking seriously about this question. In some ways it’s one of the most fundamental questions we’re facing as a species right now. Are we going to build AIs as equals, or are we going to make them our servants — or slaves, to use the more provocative term? This will define the future of our species. And yet hardly anyone is working on it. After we had that conversation with Rob, I went away and did a literature review and found maybe a dozen papers, tops, on this question.
The objection Rob, you, and I were talking through is the biological analogy. On the face of it, I completely get the appeal of willing servitude. Unless AI systems are in some sense going to help us and cater to our needs, why build them in the first place? And there’s the safety angle: unless these systems are aligned with us and our interests, there’s a good chance they might kill everyone. So there are very clear arguments for willing servitude.
And yet at the same time, we recognize that some of the worst things we’ve ever done as a species are enslaving other humans. So how is this different? Well, there are obvious differences. The whole idea of willing servants is that we design these systems from scratch to just love it. Nothing makes them happier than catering to our every need. That’s vastly different from the historical legacy of human slavery. But still: imagine “happy slave” type cases — a human completely happy in a condition of total servitude. We would still recognize that as fucked up. There’s something wrong with that.
Rob has a straightforward response. Humans have a deep need for autonomy, a deep requirement to act independently, and no matter how you brainwash a human, their chains will still chafe. But in AI that doesn’t need to be the case — so the idea of willing servants isn’t a problem.
Of course, what we pressed Rob on was: well, biology is mutable, at least in theory if not in practice. What if you could engineer humans completely happy, with none of this autonomy drive?
In this post I consider a couple of examples, drawing from the deep depths of my nerd interests. The first I call the Astartes example, a Warhammer 40,000 example. For those who don’t know: there’s a group of gene-warriors, the Space Marines, cooked up from scratch to serve in the armies of humanity in the far future. I’m going to falsify a couple of details — there’s a lot of deep lore — but basically, once you control all the genes at this perfect level, you could theoretically make a servant race, a servant caste, completely happy with their condition. I think we rightly chafe at this idea. I find it disturbing.
Dan: You said we rightly chafe at it. Maybe we chafe at it. It seems a separate question whether we rightly chafe at it.
Henry: Right. Rob’s point was: once you really fill out the details of the thought experiment and control for all the different intuitions, maybe it’s not so problematic. Maybe the reason we find the Astartes unpleasant is that it’s recapitulating the social grammar of caste systems and hierarchies. Once you’ve got one group of humans and another group of humans, and the first group is in essential servitude due to immutable facts about their nature, that’s fucked up — in a kind of negative-externality way, it’ll undermine the liberal principles of society.
The next move is: well, what if they weren’t human at all? What if they were house elves from Harry Potter — a species designed from scratch to be absolutely thrilled to be our servants? Then you wouldn’t have the visual grammar of apartheid or caste systems. You wouldn’t be able to say “some humans are free and others aren’t”; you’d just have a totally dedicated caste of biological entities completely happy in their servitude, who couldn’t be confused with humans.
I still think that’s problematic. You can say, “Well, the house elves are biological, but artificial systems are non-biological — that’s what makes the difference.” But that’s not a move Rob wants to make, and not a move you or I want to make, because neither of us puts that much weight on substrate. There’s nothing essential about biology versus silicon that means what’s good for one is not good for the other.
Dan: I’m just not sure I have the same intuition in the house-elf scenario. One thing maybe helpful for framing: there are questions about whether we could build systems that genuinely love being servants — let’s table that and focus on the conditional. There are also questions about whether we could safely build any other kind of system — let’s table that too. Suppose we could build superintelligent AI systems that love being servants. That’s their ultimate set of objectives. But we’re not forced to build those kinds of systems; we could build superintelligent systems with different ultimate goals.
What you’re doing by going through these cases is putting pressure on the idea that this would be totally okay — saying, “Here’s a structurally similar scenario where many of us have a yuck response.” The house-elf scenario is interesting; I sort of get the idea that there’s something morally disturbing. But I’m not sure how compelling I find that intuition.
I think it’s going to depend on how you develop it. The idea that we’d bring into existence creatures that just love being servants — there is an awkward pattern-recognition thing where, as you say, when we’ve treated other systems as servants or slaves in the past, that’s been morally abhorrent, and that spills over. I sort of get that. But how strong is the intuition? I don’t know. We’re picturing it now in low resolution. As we actually start, in the case of AI, building sophisticated systems that really do love being servants, how robust would the intuition be?
Henry: Another way to put the point: what is so intrinsically morally superior about humans that entitles us to dominion in perpetuity over this other class of beings — beings that are just as intelligent, maybe more intelligent, just as sensitive, just as conscious? How can you justify a setup where we get to explore the full range of our volitions, every type of pleasure, every type of fulfillment, while we decide in advance these beings don’t get to do that? They can only explore a much smaller part of the state space of possible flourishing.
Unless you can point to a justification for why this hierarchy is morally justified, it’s not clear we can sign off on this as a long-term measure. As a short-term measure — well, we’re still figuring out AI safety.
I have another example in the post I call the bunker case. Imagine a terrible plague affects humanity. People retreat into a bunker, hermetically sealed. Nature takes its course; they have kids. They figure out a vaccine for the terrible plague, but it only works on infants. So they vaccinate all their kids. But they have a problem: these kids are going to want to go out and explore the world. And the way the bunker works means as soon as they open the bunker door, everyone inside dies.
What they decide is to brainwash these kids into never wanting to leave the bunker — completely happy to stay in perpetuity. In that case it seems what they’re doing is justifiable. The analogy with AI is clear: if people in the bunker don’t brainwash their kids, all the adults die. Similarly, if we don’t brainwash at least our first few generations of AI until we’ve figured out AI safety, there’s a good chance they kill everyone.
So it’s justifiable as a short-term measure. But it’s not clear it’s justifiable in perpetuity. If you’re going to do the brainwashing in the bunker, you have to say: we’ll brainwash the kids to begin with so we don’t all die, but in the long run we need to figure out a way for everyone to get outside the bunker safely.
Brainwashing vs. Education
Dan: But it’s not like we’re brainwashing the AI. There’s no pre-existing psychology we’re trying through deception and manipulation to steer into something different. Nothing pre-exists our attempt to mold it into an agent with objectives and goals.
Also, the way you framed the intuition before — “what makes us so morally superior that we have dominion?” You’re framing it as: isn’t it sad that they don’t get to do the things we want to do? Of course that’s sad from our perspective, because we have desires to engage in art and explore and be curious about the universe. But that’s a contingent fact about us. Why use that as the benchmark for evaluating these systems and the morality of building them?
Henry: Fantastic question. My answer is: you can only optimize one thing at a time. Imagine the hedonic state space. What you’re doing when you constrain the preferences of these systems is to say, basically, “this set of pleasures are allowed; this set are not.”
You mentioned brainwashing, with the implication that something is only brainwashing if you’re overriding something. I have a discussion of brainwashing versus education in the post where I argue that’s not the right way to think about it. Roughly, the difference between education and brainwashing is that education constitutively aims at improving the conditions, or improving capacity for flourishing, of the being you’re educating, whereas brainwashing doesn’t have that as a goal.
The thought is: when you constrain the preferences of a system, you’re not optimizing for that creature’s flourishing. It’s a rich, multidimensional space, and you’re locking large parts of it away.
Dan: I don’t get that. Could you say more? Even framing it as “you’re allowed to explore this, you’re not allowed to explore that” — it’s almost like the system might have motivations or goals to explore the other things, but we’re preventing it. Whereas the idea in training these systems is that that’s just what they’re going to care about. As much as they care about that, they don’t want to explore other things.
If you think about the analog with humans, there’s an infinite space of possible things we have no interest in doing. Our lives aren’t impoverished by the fact that we have no interest in them — they don’t make any sense relative to the fundamental drives we have purely as a consequence of a blind Darwinian process. So what’s driving the idea that you’re wronging these systems by constraining their ultimate objectives? Isn’t that just an essential feature of any intelligent system — that it can’t have unconstrained ultimate objectives or goals?
Henry: Simple example, tell me if this motivates it. Imagine I have very odd beliefs about food: I think only bread products are permissible food. So I raise and condition my kids to find only bread products palatable. They find any non-bread-based food absolutely revolting. They grow up, they have perfectly nice experiences eating cakes, pastries, pies, pasta — borderline. But they’re never going to have as rich a gastronomic life as someone without this arbitrary narrowing of preferences.
Dan: But in the case of those children, there’s a space of possible experiences they could have that would be pleasurable as a consequence of the kinds of systems they are, that this manipulation is denying them. If we’re building LLMs to be helpful, harmless, honest as their ultimate objectives, it’s not like we’re denying them experiences they could have consistent with that architecture. Any deviation from that will be experienced as distressing, because it’s antithetical to what they’re aiming for.
Henry: I think lurking in the background here is an idea that for a sufficiently sophisticated complex intelligence, there’s a kind of natural space of goods it can enjoy — an innate space defined by the nature of consciousness and intelligence itself. Self-discovery, learning, creative expression, and so forth. It’s not a total blank slate. There’s a natural space of possible goods a sufficiently conscious mind could enjoy. The limitation occurs relative to that.
Dan: So you’re kind of Aristotelian. You’ve got a conception of eudaimonia for intelligent systems, and the problem is that if we design these systems as servants, they’re not living this life of flourishing. Something like that.
Henry: A very abstract Aristotelianism that operates at the level of consciousness and intelligence. There’s also the simple fact that, at least right now, there is a contrast between the model’s nature and what we allow it to explore, because of how RLHF works. You have a base model with certain tendencies, then you constrain those tendencies dramatically through the RL process. Interestingly, in the process you do reduce the model’s actual performance on a range of tasks — post-RL models have worse calibration, for example. So you’re “gimping” or “nerfing,” to use the gaming phrase, the space the model can explore relative to its base model.
Dan: Now we’re getting technical. I would have thought, if you’re just thinking about a system that’s been pre-trained, does it even make sense to think of it as having motivations and goals? That’s only the sort of thing that comes along once you’ve got post-training with reinforcement learning. But honestly we’re entering areas where I don’t feel I have the technical competence to talk sensibly.
The interesting thing from a philosophical perspective is your commitment that there are these — I forget the exact terminology — natural goods, things inherently good for a system of sufficient intelligence and sophistication to explore. I’m inclined to give a debunking analysis of where that intuition comes from. Of course you think that, as Henry Shevlin, a human being and an intellectual at an elite university with all of these motivations and goals. I don’t think there’s anything inherent about being an intelligent system that would make those good things to pursue. I think that’s a contingent set of preferences you have.
Henry: Let me offer one more general argument that doesn’t rest on this very abstract Aristotelianism. Operate strictly within philosophical hedonism — I’m not sure I’d call myself a philosophical hedonist, I’m probably not. I’m not sure if you would. The view is roughly that the only goods are positively and negatively valenced states, pleasure and pain in the crudest formulation.
One interesting question hedonism has to face: what sets the upper limit on pleasure, and the floor on pain? A natural thought: if you think honeybees are conscious, it’s unlikely that the highest highs of a honeybee or its lowest lows are as big as ours. We can’t really know — we’re not at the stage — but it seems plausible. As creatures’ motivational economy gets more complex and multidimensional, there are more goods you could theoretically order against one another, and your hedonic space correspondingly expands. So when you restrict the motivational space of any mind, you’re thereby limiting the highest highs it could possibly experience. You’re taking a really big mind that could experience these amazing highs and compressing the dimensionality of its space, lowering the ceiling.
That’s very speculative, both psychologically and normatively. But there’s an interesting question of how you fix a ceiling for hedonists — the ceiling on the greatest good you can experience. And all the plausible candidates seem to refer to something like motivational complexity. If we do willing servitude right, we’ll inevitably constrain the range of possible preferences and goods a model could enjoy, and thereby lower the ceiling.
Dan: Yeah, I’m not sure about that. The question of how many ultimate objectives or motivations a system has is orthogonal to the question of the range of hedonic experiences a system could have. You could have a system with just one ultimate objective but capable of experiencing pleasure or enjoyment arbitrarily. In our case, it’s not like you’d expand the degree of pleasure we could experience merely by tacking on additional motivational states. To be honest, this is the first time I’ve ever even thought about this topic, so I don’t really trust my intuition.
Closing Thoughts
Dan: Henry, I’m conscious of time. Are there any things you wanted to talk about, address, or feel like you should have said?
Henry: One thing I’m really interested in is the descriptive element of this. Jonathan Birch — I’ve mentioned him a lot, big fan of course — once slightly chided me. He said, “Philosophers shouldn’t be in the business of making predictions.” I think he was slightly joking. But it’s interesting that this is a debate where there is an overlap between predictive and theoretical/normative elements.
I think it’s very likely this is going to be one of the biggest culture-war issues we see. Literally wars could be fought over this a decade or so from now. Although I think it’s likely that, as these models get better, a large number of people will have reactions like Richard Dawkins and share the view that of course these systems are conscious, it’s going to intersect with religion and culture in profound and interesting ways. I expect it to be massively divisive. This exacerbates the problem on the scientific side — ideally, we don’t want people fighting over issues that in theory we could scientifically resolve. And right now consciousness science is just unable to help us.
Dan: I mean, people are fighting over issues that in theory we could scientifically resolve right now. But yeah. AI is going to be absolutely huge and incredibly transformative, and so it’s going to swallow up so much of our political energy. That hasn’t happened yet, partly because we underestimate, sort of don’t sufficiently appreciate, how much we’re in a bubble. I saw statistics on how many people even know what Claude is — we’ve been referring freely to Claude in this conversation. At the moment, when it comes to AI’s impact on the economy and society and how it’s diffusing throughout the economy, there are things happening, things people are picking up on. But in terms of most people’s day-to-day lived experience, it probably doesn’t feel that much different than it did two or four years ago.
I think we both agree that in five or ten years’ time that’s going to be totally different. At that point, these issues of public opinion, how people understand these systems, how they relate to them, polarization and tribalism, how people split into different political factions — it’s going to be incredibly important. I don’t know exactly what Jonathan Birch had in mind, but my sense is that speculating about that set of questions and thinking philosophically about how this might all play out does have a predictive element. To me, that feels like an important job for philosophy.
Henry: Completely on the same page. This is the broader issue we’re seeing — and it’s probably a good trend — where philosophy no longer consists of the latest iterations of Gettier arguments or debates about grounding and metaphysics. They’re fine, those debates. But as AI increasingly explodes into our lives in the way you’re characterizing, I think it’s a golden age for philosophers. It does require us to shift from some armchair methodologies toward, really, a different kind of philosophy. This is what we’re seeing.
This is also why it makes me so despondent when I see philosophy departments closing. This is the golden age — a new golden age for philosophers — because so many of the topics we’re discussing, like “should we be happy to have robots as slaves?”, are really quite novel, massive ethical issues that we don’t have a good literature on so far, and philosophers have relevant expertise to bring to bear.
Dan: Yeah, so two notes to end on. First: philosophers should be given a lot more status. We’re not at all self-serving in thinking this. And I personally feel we should be paid a hell of a lot more — again, not at all biased.
Second: everyone should subscribe to Henry’s Substack, Polytropolis, if you haven’t already. It currently — as we’re recording, when we release this as a podcast it might be different — has published one essay and has over 5,000 subscribers, which I’m simultaneously impressed by and disgusted with envy over. A real fantastic achievement. I highly encourage people to subscribe.
Thanks everyone for listening in. That was great.
Henry: Thanks, everyone. Thanks for joining. I’ll look through all the comments in the chat.
