Sand Gods, Foom, and p(doom) (with Séb Krier)

In this episode, Henry and I speak with Séb Krier, who previously led frontier policy development at Google DeepMind and is a Senior Research Fellow at the Cosmos Institute. He recently joined the DeepMind Institute, where Henry works as well. (Séb and Henry both speak in a personal capacity, so any hot takes are entirely their own.)

I wanted to have Séb on the podcast not only because he is one of the most insightful people thinking about AI, but also because he is an unusual voice in the broader AI debate.

On the one hand, he is very “AGI-pilled”, in the sense that he takes seriously the prospect of extremely transformative AI emerging over the coming years and decades. On the other hand, he rejects many popular narratives about the shape and nature of that AI revolution.

Among other things, we discuss:

• What Séb calls the “Sand God” view of AI, and whether metaphors of AI as godlike, alien, or “grown” help or mislead.

• Why Séb thinks of AI as a resource to harness rather than a creature, drawing on Eric Drexler’s work.

• Why he thinks recursive self-improvement is better understood as an industrial process than as a route to “foom”.

• What the Hugging Face incident does and does not show about alignment.

• Why institutions and scaffolding still matter even if we can build benevolent AI systems.

• Whether the bigger danger is misaligned AI, powerful AI companies, or concentrated state power, and whether AI is an inherently centralising technology.

• The anxious cultural backdrop to AI discourse, and what is wrong with p(doom).

• AI consciousness, on which Séb and Henry disagree.

• Why Séb thinks “pacing the frontier” is too vague to act on.

Links and further reading

• Séb on X and his Substack, Technologik

• Of Swarms and Sand Gods (Cosmos Institute, 2026)

• Musings on Recursive Self-Improvement (Technologik, 2026)

• Coasean Bargaining at Scale (Cosmos Institute, 2025)

• Distributional AGI Safety (with Nenad Tomašev and colleagues, 2025)

• AGI, Governments, and Free Societies (with Justin Bullock and Samuel Hammond, 2025)

• 20 takes about AI (X thread, October 2026)

• Eric Drexler, Reframing Superintelligence: Comprehensive AI Services as General Intelligence (2019)

• Richard Danzig, Machines, Bureaucracies, and Markets as Artificial Intelligences (2022)

• Tom Reed, The Goodhart Singularity (2026)

• Alexander Lerchner, The Abstraction Fallacy: Why AI Can Simulate But Not Instantiate Consciousness (2026)

• Our earlier episode, Steven Pinker on AI Doom and Consciousness

Transcript

(Please note that this transcript is lightly AI-edited and may contain minor mistakes)

Introduction

Dan Williams: Welcome back. I’m Dan Williams. I’m back with my friend and co-host, Henry Shevlin. And today we are joined by Seb Krier. Seb previously led Frontier Policy Development at Google DeepMind and recently joined the DeepMind Institute, where Henry works as well. Seb is also a Senior Research Fellow at the Cosmos Institute. He speaks today in a personal capacity, so any views or hot takes that he delivers as part of this conversation are entirely his own. That’s also the case when it comes to Henry.

I wanted to get Seb on the podcast not just because he’s one of the most insightful people thinking about AI, but also because I think he’s a very unusual voice in the broader AI debate. On the one hand, he is very AGI-pilled, in the sense that he takes seriously very transformative AI emerging over the coming years and decades. On the other hand, he rejects many popular narratives in terms of how people often understand the shape and nature of that AI revolution. So welcome to the podcast.

Seb Krier: Thank you very much. Happy to be here.

Henry Shevlin: I would also add, Dan, that a great reason to have Seb on the show is his phenomenally good Twitter game, which specifically consists of some brilliant long-form essays. I think I do a pretty good game myself, but it’s mostly hot takes and K-pop tweets and memes. But Seb offers some of the best long-form, really thought-out essays on AI Twitter out there. So I strongly encourage everyone who’s not a follower of Seb already to get on there.

Seb Krier: Very kind, Henry, thank you. But it is all just slop, just human slop.

The sand god view and AI metaphors

Dan Williams: It is definitely not slop. I think Seb is the leading public intellectual of AI on X slash Twitter. Okay, so Seb, you’re very sceptical of a certain picture of AI that is very influential in some circles that you call the sand god view of AI. What is that view?

Seb Krier: It’s very hard to condense or summarise it, not least because I think different people within that ecosystem have different views themselves. But I think what I mean by that sand god view is a few things. The first is the god element, where people talk about AGI or ASI as something that’s quasi-omnipotent, omniscient. Essentially, you could substitute the word ASI with God in the ways people talk about this, and it would just work all the same.

And so I think when that is the case, then it gets very hard to argue anything because people always have this eternal checkmate of, “Well, the sand god could do X, Y, Z, so why would your argument even apply?” So that’s the first thing. It’s a bit of a weird way to talk about ASI.

The second is the ontology that people use for thinking about intelligence, models, what they do, what they are. The frames are often talking about them like they’re aliens, they’re creatures, things that we’re growing, not building, and so on. And so I think the ontology leads people to just think, “Well, how do we coexist or cohabit with these new entities that now live on this planet with us?” And I think that’s a mistake.

And then thirdly, the obsessive focus on the singular. Solving the one alignment problem, the singleton, the winner-takes-all model of economics. People talk about the model or the AGI or the ASI. It’s always “the”. And I think, for some reason, this way of thinking about AI seems to lead people towards Malthusian or authoritarian or paternalistic views sometimes. In reality, I think we’re going to get all sorts of agents, AI systems out there. And that’s a feature, not a bug.

Henry Shevlin: I think I agree with the first and third part of your view here, Seb. The idea that ASI is probably not going to be godlike or straightforwardly omnipotent, it’s probably not going to be a singleton. But I’ve always really liked this metaphor of growing AI.

And I think it’s really helpful, particularly when communicating to the public, that we don’t always easily anticipate the features that models can have, that what you do when you’re building an AI model is set constraints to the training environment, which I think is accurately captured by this metaphor of growing. So I’m curious why you push back against that.

Seb Krier: Partly because I think “training” works fine too, to talk to the public. I think they get a sense that training implies a bit more agency and choice in the process, whereas growing, I think, less so. I understand why people do it, right? It’s to explain that you’re not deterministically coding everything and designing. And I think that’s a useful thing to communicate.

But on the other hand, if I grow a plant, it just becomes that plant. I don’t have much choice in what it becomes. I can water it. I mean, there is some choice, but that’s usually not what people associate with the growing analogy. Whereas I think with training systems, you get to ask, “Well, what are you training them to do, and why, and how?” And so on. And I think these are the questions that matter because they open up the space of possibility and the agency we have.

But I think growing per se isn’t the worst metaphor. It’s not just the growing thing alone, it’s everything else that comes around it, right? It’s like the characters as little aliens, and they’re so mysterious and spooky, and we need to coexist with them. It’s all that side of things I sometimes think over-indexes on a bunch of other metaphors and views.

Henry Shevlin: So maybe part of this comes down to how horticulture-pilled you are. My wife is a really avid gardener, and she grows a lot of roses. And the thing she does to these plants is pretty extreme. She has a huge amount of agency. So when I hear people growing, it’s less like, “Toss a seed in the ground.” The image that comes to mind is something more like a bonsai tree or a complex rose bush. So I don’t know, maybe the metaphor lands differently depending on how you think about the horticulture side.

I was just going to say I actually also quite like the alien side, up to a point. Take something like the shoggoth meme, which I’m a huge fan of. I’m guessing you’re not a fan.

Seb Krier: Not a huge fan. It’s useful to communicate certain things. But with most metaphors, I’m as concerned with the ways in which they don’t apply. And I think that’s the bit that people usually throw away or assume away. But I think the shoggoth is broadly fine. There’s a lot of different masks. But it also implies a few things, right? It’s like a big, scary alien behind the thing.

It’s very mysterious. We don’t know. It’s hard to control. The mask is also seen as extremely thin, which, in some sense, it is; in other senses, it’s maybe not. But the shoggoth thing isn’t the worst part, or at least it’s not the thing I’m most concerned with, I guess.

Dan Williams: I’m 100% with you on this, Seb. I think these metaphors—the idea that it’s grown, not designed, we’re ushering into existence this new species or alien life form—I think if the only thing somebody knew about AI came from these sorts of talking points and metaphors, and then they actually encountered the way in which AI systems work and how incredibly steerable they are, and how incredibly good they are at following people’s instructions and intentions, I think they would be baffled. Which I think suggests that there’s a huge mismatch between what these sorts of metaphors convey and the reality of the technology.

And I think that they often get used precisely as part of this broader worldview, where the idea is we’re in some sense bringing into existence some kind of adversary. It’s going to be, by default, misaligned and have drives and goals of its own. And I do think that’s very misleading for some reasons that you touched on, Seb, and we might get into in greater detail.

I just want to start with the first thing you said about how often the way people talk about AGI, but more specifically ASI, artificial superintelligence, it’s like they’re imagining a god. This also came up in our conversation with Steven Pinker, where he also thinks that one problem with the standard doomer arguments is what they’re really imagining when they talk about a superintelligence is a system that is omnipotent and omniscient and so on.

And I think, on the one hand, I do sympathise with that critique. And I think it is a problem that imagining arbitrarily capable systems is like a get-out-of-jail-free card in debates about AI. But I think, on the other hand, what the response would be is, “Look, we’re not imagining systems that are literally godlike, literally omnipotent, literally omniscient, but we should allow for the possibility of systems that are just much more capable than even the most capable human beings across all possible domains.”

And I think some people who see this whole domain very differently from you would say this doesn’t register in the way in which you’re modelling the future. This possibility—and there’s lots of uncertainty about exactly how capable these systems can get—but this possibility that we could have just stunningly capable systems. What do you think of that kind of response?

Seb Krier: I don’t particularly have an issue with the capabilities side of things. That’s priced in, that you will get systems and agents that can do all sorts of very impressive things, better than most humans or individual humans and whatever. It’s more that you also have all sorts of other affordances available to shape and limit and constrain these capabilities, and to make them useful to you, to make them work in a particular way. And it’s the emphasis on this latter part that I think is missing from the usual sand god picture.

So I’m totally fine with imagining systems that are doing very, very impressive things. Like an AlphaFold, but for drug discovery and a bunch of other things, and so on and so forth. That’s not really the issue. It’s more that the god analogy does a bit more than just say it’s going to be very, very capable.

You usually do not think of gods as things you harness and limit and constrain and bound, and work for you. I’m sure someone somewhere’s done that. But in general, that’s not how people think about deities. So I don’t have a problem per se with thinking of really, really advanced capabilities.

In fact, a view I like a lot more than thinking of them as deities is based on Eric Drexler’s work, this comprehensive AI services type work, which thinks of these as hyper-capable services, or intelligence as a resource rather than as a creature. And I think this is a different set of analogies and metaphors that, in my mind, better tracks what the technology is and where it goes.

And I know people think this is cliché or whatever, but I think even corporations are an example of a form of superintelligence. And there are many ways in which we constrain and direct and shape and align and limit corporations. In fact, there’s a really good paper by Richard Danzig—I think he was former Under Secretary of the Navy—looking at machines, bureaucracies and corporations, and the ways in which they are alike. But also, in many other ways, they’re not, right? I think these analogies only work to some degree. But I think the Drexlerian view is closer to how I think about it than the alternative I just outlined.

Henry Shevlin: So just two thoughts on that. I’m curious if there is an alternative metaphor besides corporations for thinking specifically about beings with a greater overall intelligence compared to humans. So another metaphor that’s often used, and I’m curious what you think of this one, is chimps versus humans, with AGI or ASI occupying a similar higher role. So I’m curious if you have any metaphors specifically for that.

The other thing I wanted to pick up on—but maybe I’ll let you answer that first—I did want to dig down into Drexler’s CAIS a little bit more. But go tell me what you think about metaphors.

Seb Krier: Sure. Well, I think the first thing is that you talk about intelligence in there, and already I prefer capabilities to intelligence, even if, effectively, they probably track each other to some degree. And I don’t really like the idea of thinking about intelligence as a scalar. You start off with, I don’t know, bugs or something—the video’s a bit buggy—but you start off with bugs, and then you’ve got chimps, and then you’ve got humans. And I think this is again very close to the creature view of AI and AGI, which just says that as soon as something is more intelligent than you, then you’re probably screwed. Not necessarily, but—

Whereas I think that if you think about them in terms of services or as a resource to harness, you don’t need to get rid of the super capabilities or whatever. But you also reify, or at least assume, some degree of control and shaping, which I think is in fact what we have. I don’t know if I’m answering the question, but I don’t really like the chimps and humans.

I’ve always found it funny, anecdotally, in some safety discussions previously: “Well, the AIs might deceive us or something, so we’ve got to catch them whilst they’re not as smart as humans, but just right underneath where we can still catch them.” And it’s like, all right, so the people saying this are effectively saying that the model is solving Riemann hypothesis type mathematical problems and whatnot, and they’re just slightly below their intelligence. They’re still able to catch them. But as soon as it goes over—it just feels like a weird way to conceptualise both models and intelligence. So it just isn’t really something that appeals to me.

Henry Shevlin: So again, I find myself agreeing 60–70% with you. I certainly am unsure about the massive, rapid leap that people make from superior-to-human capabilities in all or most domains to, therefore, game over, humanity. That seems way too fast for me.

On the other hand, I do worry that unless we have some kind of metaphor to capture the difference in scale of capabilities between humans and potential future AI systems, then a lot of people just don’t take certain possibilities seriously enough, right? I’m sure we’ve all heard the bad AI safety takes: “Well, you could just turn it off. You could just unplug it.” And I think something like a god metaphor or a human-to-chimp metaphor can be useful for capturing, “No, no, you’re dealing with an agent that can probably outthink you in most of the relevant domains.” So I do feel we need something there.

Seb Krier: It’s an interesting way to put it, because I was going to ask, do we actually need a metaphor? Can we not just talk about AI as it is? And you mentioned, “Well, we need people to take it seriously.” So there’s a bit of a normative or at least an advocacy-ish thing here, where it’s like, “Well, people don’t take it seriously enough, and therefore we need to find a way to wake them up,” which is very often part of the assumption why people do certain things and describe things in certain ways. And I’m less keen on that.

It’s like, well, actually, we can just describe it as it is, and people will also adapt as they try things and find out more about what these things do. And I’ve heard this for a decade now or so, of, “People don’t see what’s coming. Everyone’s dumb, so we need to find a way to freak them out.” Again, I’m caricaturing a little bit here.

Maybe there are some better metaphors out there that are neither corporations nor creatures, but I haven’t really thought of one, and I guess they’re very hard to come by. Did we need metaphors to talk about companies and calculators and computers and stuff? How did people talk about computers? Computers are extremely powerful and just changed the world in many, many ways in the nineties. Did we need to find ways to describe them to people? It’s not clear to me that this is necessary, I guess.

General intelligence and specialised services

Henry Shevlin: I definitely take the point that saying we need a metaphor to sufficiently motivate people is building in a ton of normative assumptions. So, very last thing, and then we can move on. I wanted to double-click a little bit on CAIS and Drexler, because I was a fully paid-up believer in CAIS. The idea that, rather than having true general-purpose AI, we have these specialised superhumans in individual domains. You’ve got your amazing legal AI, your amazing accountancy AI.

But I was disillusioned with this vision basically post-GPT-4, once it really became clear that probably the best way to build a top accountancy AI was also the best way to build a top law AI and a top writing AI, which was a general-purpose large language model. So I’m curious if you had a similar shift, or whether there are bits of the CAIS model that you hold on to and which you reject.

Seb Krier: CAIS was written in, what, 2017 or 2018 or so. In some respects, it’s not fully up to date, of course. But it’s not really critical in my mind. Drexler did maybe, at the time, over-index or talk about things like narrow systems and translators. Machine translation was an example he used a lot in the text.

But I think what matters more than that is the general properties he ascribes and everything else he says there about AI, about the nature of power, the nature of intelligence, the nature of engineering and so on. I think all of these things continue to apply. And even with general models like GPT-4 or 5 or whatever, I still think that the specialisation side of things applies. He couldn’t have seen it at the time, but I think right now we fine-tune models to be good at particular things, or better at certain things. We scaffold them in particular ways to make them useful to a particular set of use cases and users in context.

And so there is an effective way in which we’re basically seeing these comprehensive AI services. Now, they’re built on what are general-purpose-ish models that are very general in nature. But I don’t think everyone out there is just using the base—or at least not even the base model, but even the instruction-tuned model—of these generalist models. Everything that people use is, firstly, a model that is fine-tuned and specialised and shaped in a particular way. And not even just that, but a scaffold, a system, software, affordances, tools and so on that essentially end up being a particular specialised service.

A bit like how we see with humans, right? We are all general intelligences here, but each of us does something slightly different and answers to different people and stakeholders, has different sets of skills and trade-offs. Certainly, I think the analogy with humans doesn’t actually work that well. In fact, that’s my meta point. But I see today—in fact, what I think will continue to happen—is more of that specialisation. Realising that specialisation is a feature again, not a bug. And so is division of labour, and so on and so forth.

I don’t think we are, nor that we will be, using one big system, or small system, whatever, that just does everything and anything. I think of models and intelligence more as a resource that we’re going to harness and leverage to translate intelligence into utility. And that part is where I see all sorts of companies, start-ups, essentially whoever, just providing these services, right? What we’re seeing with Harvey and all these other companies, I think, is not just a temporary thing.

In the same way as, a few years ago, people were saying, “People are just building wrappers, but the next model’s going to eat everything.” I was like, well, not really. I think that depends on what kind of wrapper you’re building. If you’re building a thing that just adds a 2% boost on some evals, of course. But there are also the scaffolds and harnesses that aren’t so much about just capabilities per se, but more about utility and transforming this resource into something useful to people.

Dan Williams: And in terms of just zooming in on exactly what this scepticism consists of and what motivates it. So there’s an idea, I guess, that we are on track to build these unified AI systems that can simply do everything, at least when it comes to cognitive abilities.

And one reason you might be sceptical of that is, as you’ve mentioned, the economics of how this plays out. Even if there is a system which could in principle do everything, this generalist system, that wouldn’t necessarily mean that it would be advisable for everyone in their specific use case to use the biggest, most generalist, most capable system. They might benefit from using more specialised systems that would be more cost-effective.

But another kind of view is just sceptical of the whole idea that intelligence ultimately is going to come bundled, or packaged rather, in the form of a generalist system that can in principle do everything across all possible domains. And I take it some people who are sceptical of this notion of a unitary generalist AI are just sceptical of that whole concept of intelligence.

And then I think in response to Henry’s point, which is, actually, maybe that kind of scepticism made sense five or ten years ago, but it looks like now, with frontier AI systems, they do have this really generalist competence profile. I think you might still argue, actually, the fact that these systems, even though they are superhuman at some things, the fact that they’re so jagged in terms of their competence profile suggests that they are a distinct kind of intelligence rather than being on the path to a general-purpose intelligence. So which of those two ideas, or maybe it’s both of them, is driving your scepticism here?

Seb Krier: I’m probably on your side here, in terms of being a bit sceptical of this unitary generalist. And in fact, that was what Drexler was arguing against in his text: there’s actually not that much demand for some sort of thing or entity that just learns everything from first principles. And I think it’s more interesting to look at this as different kinds of intelligences, as you say.

In fact, one thing I remember Drexler writing in there was that I think Bostrom was saying something like, “We have to zoom out and see all the different types of minds that may potentially exist.” Whereas Drexler was saying, “No, you’ve got to zoom out even further and see there’s many different types of intelligences out there. Minds are just one small part of that.” And so I think that appeals to me.

The systems we do have today are general in some respects, but very jagged in others. I think the jaggedness, of course, might improve over the years. And the nature of the jaggedness is relevant here. I don’t have a very clean or clear way of separating what I think of as the kinds of problems that get fixed with a bit better data and buying some better RL data sets or something, versus the kinds of things that are just more inherent to the type of problem you’re trying to tackle. Something that’s more like Hayekian local knowledge, or something.

So I don’t have a very, very clean way of separating these things out. But it seems odd to call models as they are today this capital-G general intelligence type thing as was conceptualised ten years ago or so. At the time, in fact, the debate was on something with pure RL, AlphaZero-style, that would just, through trial and error, through exploration of different paths, build up this knowledge and competence and intelligence from first principles. And I find that a lot harder to believe.

Again, maybe that is a thing that is feasible in principle. It just doesn’t seem to be the thing that we’re doing in practice. But it could well be that once you do have these superintelligent services that I’m describing, and this generally massively enhanced technological R&D ecosystem across the world, at some point you’re also able to create these more biology-like intelligences. So it’s not completely off the table. It just doesn’t seem to be the systems we’re building today.

What current models can and cannot do

Henry Shevlin: So not to put you too much on the spot, but just to give an example here, what’s your clearest case, or what’s the simplest cognitive task right now that current-generation LLMs can’t do or can’t do effectively?

Seb Krier: I always struggle with this question because I don’t really know where to exactly—I mean, there’s certain things. The classic: if I have an intern, he’s not super useful on day one, but after day thirty or after a few months, he understands me, my preferences, things much better than I think models do, and is able to adapt much more fluidly, I guess, than the models that are a little bit more janky.

But a very basic example is just using my music software, Ableton, and going on and creating some nice music with it or something. I think, again, they will get there, right? It’s not some sort of inherent limitation. You will get someone creating the Ableton data set, and you’ll train it to be good at that, and then they’ll just do more of that. But you’ve got to essentially train it on that and make it good at that too.

So I don’t really know. I think there’s plenty of things where I’m like, “The models aren’t that useful.” I don’t like the writing style of Claude. In a weird way, the limitations are maybe less about what it’s capable of doing or not, and more about how it’s doing it and what I’m actually after in a task or something. It’s very hard to get a very clear answer here.

But plenty of things of normal usage of graphical user interfaces and all this stuff. But again, these are not things I want to say, “And therefore...” This is just like R’s in strawberry, right? I think it’s the kind of thing that will, over time, get ironed out. So capabilities isn’t really the thing I’m usually unsure about, I guess.

Dan Williams: And I think this is sometimes thrown out as a gotcha for people who are highlighting deficiencies or things that the current AI systems lack. Not to suggest that Henry was setting this up as a gotcha, but sometimes some people do throw it out as a gotcha.

And I think there are some things you can say that are quite precise about what AI systems lack in terms of continual learning and data efficiency and certain forms of creativity that they seem to lack. But I think there’s also just this general sense that even if you can’t precisely say exactly what it is that they lack—and after all, why would we expect us to be able to say this in a very precise way? Because we’ve only had these systems for a very short period of time.

And I think this task of really distinguishing what it is that human beings have and they lack, and vice versa, that’s a difficult task. I think we shouldn’t expect us to be able to do that necessarily spontaneously. But the mere fact that you can’t do that, I think, shouldn’t be taken as evidence that therefore the systems are nearly there. They’ve reached this point of AGI.

Seb Krier: Well, if anything, just very quickly, I think there’s something also along the lines of: there are certain things the systems are capable of doing in principle. They’re just not set up to do it for me. And so this is where the services thing comes back. The reason I might use a particular type of scaffold is because, great, someone’s actually designed a thing that checks laws programmatically or something, and then checks case law or whatever, and it does this whole legal pipeline, I guess, that leverages and uses and harnesses this intelligence in a way that gets me what I actually want and need.

And so this is not a capabilities thing. It’s more about: you’ve got the general intelligence now, you’ve got to make it useful for me. I can’t just ask a model that’s made for Q&A, chatbot-type stuff, a question that depends on access to a lot more information elsewhere and a structured way of thinking about a problem. There’s things it’s not really about what they can do. It’s more about what they’re set up to do or something. I don’t know.

Henry Shevlin: I was just going to chime in that I wasn’t intending it as a gotcha, as you noted, but I don’t think it’s an unreasonable critique or challenge to say, “Hang on, if these models are so bad...” I know that’s not what you’re saying, Seb, sorry.

Seb Krier: Well, that’s the point, right? Because I’ve never said anything about capabilities, right? And it’s like, “Well, what can’t they do?” It’s like, “Well, I didn’t say they couldn’t do things.” It’s not the argument.

Henry Shevlin: I guess maybe what I’m pattern-matching on is there certainly was a type of AI critic that is not you who spent a long time—I’m thinking of the people who pointed to, “Look, the model can’t count the R’s in strawberry,” blah, blah, blah—who would point to these quite trivial failures of models, often to do with things like tokenisation issues, and say, “Aha, you see, the models are really dumb.” And I realise that’s not what you’re doing.

Perhaps also I’m indexing on myself a little bit because I saw things like causal reasoning and social reasoning as really big bars that large language models had to overcome. And then GPT-4 did largely clear those. There was a real discontinuity in performance on a lot of the data sets I was watching. And then that was a gigantic Bayesian update moment for me when I was like, “My God, these things can now do most of the things I cared about.” So I don’t know, maybe that’s part of why I index on these: what can they do? What can’t they do?

Seb Krier: To be clear, I have had these as well, multiple times, right? It’s [unclear] how I’ve always thought about it or something. I remember, three years ago or so, someone asked me, “What’s a stupid test for when we get AGI?” And I was like, “Well, there’s a bunch of really obscure memes that I really liked at the time, which only a subculture had seen.” Explaining that meme clearly would be a cool test of understanding or something. And it’s like, wait, a few years after that, it does that really well.

Of course, it’s the quality of the data. It’s what’s in the data set. It’s what is in the training set. So now I think, at the same time, great, it can do it. It’s not impossible, it’s capable of doing the thing. And now I’m more interested in why it’s capable of doing the thing, what’s going in and out of these systems.

But I agree. And there’s a lot of those kinds of criticism or critique that, “Well, it can’t really do this. It’s impossible,” and whatnot. I think there are certain domains where there are some stronger arguments for why, say, biology, things are slightly different than they are for, say, translation or these tasks we typically use them for in digital environments, because of different properties to biological systems.

But even there, first of all, I don’t have the expertise. And secondly, it’s just easier to remain agnostic, and when it comes, you’ll see it, and it’ll work, and then we’ll find out. I understand why one would ask that, particularly to people who just highlight limitations I think will never get solved. And it’s like, well, that’s not a very future-proof position to be in, I think.

Recursive self-improvement

Dan Williams: Okay, just before we get onto the elements of your more positive picture of how we should think about AI, although you’ve mentioned some of the elements of this picture already, the final thing I wanted to talk about in terms of elements of this picture that you reject is recursive self-improvement.

There’s an idea which often gets bundled with these other views about imminent superintelligence that’s like an alien species that might be our adversary, which is: one of the reasons why this is such a dangerous situation we’re in, a state of emergency, is that, at the moment, AI systems are playing a role already in this process of designing future AI systems that, when they’re built, then play a role in designing even better AI systems. Soon that whole process is going to get automated, and then you’re going to get foom. You’re going to get this explosive process of improvement when it comes to the model capabilities.

My sense is you’re sympathetic to parts of that story, but you’re also sceptical of parts of that story. So what’s your take on RSI?

Seb Krier: I guess there’s three things. The first is that I don’t think we do, in fact, have recursive self-improvement. The second is that, as you say, my issue is more with how people use the term and the implications that come after, rather than just the expression itself. Before, they used to talk about hard takeoffs, fast takeoffs or whatever, and then before that it was foom. But really, the way people talk about all these concepts is very, very similar. And I think it’s worth unpacking the specific details.

And the third is: I think of RSI as more of an industrial process than something about a model improving itself.

The first thing is, first of all, I think there’s no self—not in a Buddhist way, but I think there’s no... Claude isn’t really... It’s a bit of a weird way of abstracting a bunch of different things to just call the whole thing self-improvement. Even the recursive side is a little bit unclear. I think a recursive function in computer science is a procedure that calls itself from within its own code. Now, I don’t want to litigate that point too much, but as far as I can tell, this is not really what we see. And it was a concept that, when Yudkowsky used to talk about seed AI, that was the idea: an AI that would change its own code with its own intelligence, just make itself more intelligent.

Now, maybe you see something like that if you zoom out a little bit and look at this whole industrial process of training models and parts of that being automated. But then, what is the self here? Who is the seed AI? It just gets a bit muddy and weird to talk about.

And secondly, I think we are seeing huge improvements in how you’re using AI to automate parts of the stack. And that’s like, right, okay, so you’re getting productivity benefits and you’re accelerating your development or pace of work. Shocking. I think that was always to be expected.

And then, even if you do—and that’s the part I’m a bit less certain about—is the closing-the-loop part. I think that the loop actually changes quite a lot, and even if you get 95% of the pipeline automated, the remaining 5% then becomes the bottleneck, or the part that essentially, in some way, slows things down. And that’s also fine. It might be by design, or it might be: actually, you need certain things that depend on serial time. And that’s just how it is. But that doesn’t necessarily mean that, as soon as you’ve automated 95% of things, things go very crazy.

And that’s the other part: when people say things go crazy, what do they really mean? Do they just mean an accelerated cadence of model development, or do they just mean more societal transformation and changes and whatnot? And I think often people talk about this stuff with the latter in mind. And this is where I’m like, well, not really. You could have a system, or at least an industrial system, that produces a model every week or whatever, but you still need all sorts of feedback loops with the real world and with users and with adoption and the financing side and making money. You need a lot of compute still, you need deployment, you need adoption. All these other things take a lot of time.

And so I just think of RSI more as a general process of productivity enhancement for technological production and invention or something.

All of that to say that, yeah, I think there is a form of RSI that is kind of an “always has been”-type meme happening, right? And we’re going to continue seeing it, which is this combined economic growth, productivity growth, technological development and stuff. This has been going on for a while. But I just don’t really see that as being something where suddenly you get a process that gets you really crazy ASIs, and they get even more intelligent, and then you get a fast takeoff or anything like that. I think this whole other side of the story doesn’t really hold.

You need to get better data for certain parts. Something I just jokingly asked a friend the other day was: if you had a fully automated, 100% loop of automated development, where exactly does the signal come from where people are annoyed at Claude-y language everywhere and want to change...? Who finds that out and realises and changes the thing internally to the models? And if you say a human somewhere, then you’ve already assumed that there’s some human involvement in the process.

Anyway, this is more of a side point. But TL;DR is, I think RSI is a fine concept. I just think the way it’s often talked about is basically analogous to just saying foom or fast takeoff and stuff like that. And that, I think, doesn’t really obtain, I guess.

Henry Shevlin: Just to probe a little bit deeper, I completely buy and am very sympathetic to your scepticism about how fast these loops are actually going to work when it comes to economic and industrial policy in the real world, where you have everything from, as you say, raising finance to getting permits to international regulations in different jurisdictions. I think seeing RSI in economic terms in the short term seems to me to flatten a lot of really complex and interesting terrain.

But at least in principle, a priori—if we’re thinking not so much about Earth society today, but we’re thinking more like an AI operating on the moon, slowly building out, Factorio-style, its research labs and its industrial production—there it seems much more plausible to me that you can get something like RSI.

So I’m curious whether it’s more like a fundamental conceptual objection to the idea that you could ever get these dynamics. I appreciate part of the critique is just how sloppily the term is used, but focusing on proper, well-operationalised definitions, like METR’s various operationalisations, when it comes to those, is there a deep conceptual objection, or more just: it won’t play out like that in our world because of all the restrictions in place?

Seb Krier: It depends. I’m trying to think about your example of a system on the moon. Can you unpack that? What happens? What does it do? What does the training essentially make it good at? And what does it get better at specifically? What traits or characteristic variables do you use to qualify this intelligence improving, if it is done outside the context of any human utility or something?

Henry Shevlin: I was just operating with something like the image of a von Neumann probe. Maybe it has a limited amount of compute, but it has enough basic resources to build copies of itself. As it builds copies, it generates energy surplus and mineral surplus, which you can then use to invest in building out additional compute, which you can use to research better ways to build, give itself either better physical capabilities, better computing capabilities. The next generation of von Neumann probes or equivalents are more efficient; they can do more complex planning. These, in turn, can feed into better research loops for planning the next generation and so forth.

Seb Krier: I guess that’s a plausible story. Maybe there are fundamental limits to the physics of this that I’m just unaware of, because it’s not my domain. So I don’t really know how fast von Neumann probes work and whatnot.

I guess I’m less interested in that angle because it’s like, yeah, well, maybe in principle somewhere you get something like that. But right now, you’re seeing people talk about regulating AI because of RSI and so on, on Earth, with the systems we have today, with the industrial processes that we have today. So I think we should separate these very neatly.

And I think certainly the community doesn’t really seem to do that. They don’t seem to say, “No, no, this is just a thought experiment for, in principle, you could imagine blah, blah, blah.” It’s like, fine, sure. But then, in terms of what is actually happening here on Earth and with labs, you get different claims, right?

Some people are like, “I only care about the fact that we’re going to get more capabilities and faster capabilities because it just goes faster.” It’s like, yeah, but that’s fine. That’s what the productivity benefits or enhancements of these systems always entailed. But sure.

Whereas when it gets closer to, “But things get crazy, and it’s uncontrollable, and what about safety?” and so on, it’s like, well, there’s answers to all these questions too. It’s not just: you automate data labelling and a bunch of training stuff, and then suddenly everything goes wild. I just don’t really see that happening.

Deployment and real-world feedback

Dan Williams: Just to do my attempt at summarising part of your critique, then, Seb: people often draw this distinction, you’ve drawn this distinction, between how much more capable the models are becoming and how quickly that’s happening as a consequence of AI systems getting involved in the process of designing better AI systems. And then there’s this other question of deployment.

And you might have the view that says, well, maybe you can get this crazy progress when it comes to just model capabilities, even if, when it comes to issues of deployment, you’ve got all of these bottlenecks and sources of friction and speed limits and so on.

But you might also think, actually, these two things are much more closely connected than that story would suggest, both in the sense that it’s only really through deployment that you’re going to be able to get the capital so that you can actually fund these training runs in the first place—you don’t have a purely standalone economy—but also, and I think this is more interesting and radical in a way, the idea that actually you’re just not going to be able to build more capable AI systems unless you’ve got deployment in the real world. Because it’s only through deployment that you’re going to learn, first of all, what it is that these systems can and can’t do, but also so that the systems themselves can get access to the data which is necessary for them to get better and better in the first place.

Is that a fair summary of your critique, and would you add anything to that?

Seb Krier: I think so. There’s definitely this signal of: what do people actually value? If you’re going to try to sell AI at scale, what does improvement actually mean? Where do you need to improve? What components are you making? And I think these are all choices you make. You get these signals through deployment very often, not just through doing R&D and sitting and being like, “Well, I suspect that people will want more architectural design capabilities or something.” I mean, you can do some of it like that, but I think broadly, even then, testing whether your hypothesis works or not is effectively done through deployment.

But the other thing I would add is that, once you’ve automated the loop of internal model development or something, you get a huge productivity boost, right? I don’t think that productivity boost implies infinite... I don’t know how to put it, but it’s like you’ve invented microscopes now and you’re just far more productive as a biologist. You can do far more work; things are just going better. But that’s it. It’s like a one-off big upgrade. But that doesn’t mean that every five seconds you’re getting more intelligent and more efficient, and suddenly this curve goes on forever and ever and ever. I think it’s more like, well, we’ve invented a bunch of tools and systems that make everything far more productive. But this productivity boost isn’t, I guess, recursive ad infinitum.

Henry Shevlin: Just one final probe on this before we move on, because I know we have other interesting stuff to cover. An objection that I’ve heard here is, well, okay, look at AlphaGo. You might say, “Look, of course you can’t get any kind of recursive self-improvement in Go, because the system would need to go out and play a lot of games.” Except no one says that, because AlphaGo can just simulate billions and billions of games.

How much of what we learn from deployment do you think might ultimately just be internalised at the simming level, and thereby quieting some of, or addressing some of, the concerns that you have about RSI, allowing potentially very rapid improvement without the need for that deployment phase?

Seb Krier: I’m a bit more sceptical about that claim because I think that Go is a very closed system. And so the conditions are there for you to play games against yourself for a very long time and learn and derive things, because you’ve got the entirety of the picture. Whereas everything else, as in the world, is, I think, far more complex and far less of an easily predictable closed system.

I guess I’m a bit more sceptical of the claim that, by just thinking really, really hard in a basement through first principles, you can just simulate Jane Street. I think a lot of that, in fact, has to hit reality in some sense. There are endogenous changes happening as a system interacts with the world, and I think that’s an important element here.

I think you can go pretty far, right? You can use models to generate datasets. That gets you pretty far. But I don’t think it’s sufficient or equivalent to saying we’re just going to get a lot of games against Go or something. It depends on the fidelity of your simulation. And even there, I’m not too sure how accurate these simulations really are just yet. There’s some benefits in creating accurate world models that are more than what you’re generating through text.

I don’t have extremely strong views here, but I just don’t really feel like the game of Go is comparable to, say, intelligence and everything as a whole or something.

Dan Williams: Tom Reed has a good article on the Goodhart singularity, where I think he presents quite an interesting argument against the idea that you could, purely through disembodied ratiocination in a data centre, through simulation and so on, acquire the relevant kinds of knowledge.

Scaffolding and AI control

Dan Williams: But let’s move on, Seb, to the positive vision you have, both in terms of how you think everything is likely to develop and also, normatively, how you think things should develop and the sorts of things that you care about.

My sense, as someone who reads your work, is: the sand god view, as you understand it, precisely because of its assumptions about AI arriving in this extremely capable form in the package of a single, unified AI system, encourages us to think that many, maybe most, of the important questions concern almost the psychology of that AI system. Your point is, actually, just focusing on the inner workings of the AI system is completely inadequate in terms of understanding how this AI revolution is likely to unfold and the societal impacts. So you’re much more, I think, interested in these questions of institutions and scaffolding and incentive structures.

Is that a fair summary? And if it is, what is your more positive vision for how to think about the future of AI?

Seb Krier: It’s not necessarily a more positive vision. I’ve got a positive outlook on the technology and so on. But I think broadly the idea here is that it’s useful, to some degree, to think about this agent-focused view of the world, or this “Is it aligned? Is it good?” The inner alignment stuff—I’m not saying it’s useless or that it’s irrelevant. I think my claim is more that it’s insufficient.

It’s one part of the picture, and then you keep on building on layers outside of that agent, or the model or whatever. There’s plenty you can do at the scaffold and harness level. And, in fact, a lot of the systems we use today are useful precisely because of all these choices made around that. And not only that, but it gives you a lot more design and optionality, or engineering possibilities, to actually create systems that are safer, or having models doing different agents, I guess, one checking the other, whatever.

The optimistic view, in a sense, is that a lot of the AI control agenda that labs are now pursuing is essentially what Eric Drexler was calling for in 2018. He was saying, well, you can actually use a multiplicity of models in different ways. You can scaffold them, you can get a bunch of systems that do not communicate with the monitor. You can do a lot more engineering, essentially, around how you design these systems, rather than having to always—which was the default view at the time—just bet on: is it aligned or is it not aligned?

I’ve always found that dichotomy a bit weird. Is it good or is it bad? And if it’s good, then what? So I think my view is more along the lines of: you get a lot of different levers of intervention. You get the model training and the personality and the character you want to train into it and the capabilities it has and so on and so forth. But that’s not all there is. There’s going to be plenty on the engineering side for the scaffold, on even laws outside of it.

And in some sense, I think of institutions and scaffolds as very similar things, ultimately. It’s placing boundaries and creating incentives and limiting certain things. So there’s many, many different layers that influence the outputs and the safety and the properties and the characteristics of these systems we are building. It’s basically just my general vibe, I guess, which goes against the view that the main thing that really matters is the model’s alignment, and then everything else is a bit of a boring side thing. Maybe that’s one way I’d put it.

But the positive vision is: the only thing this gives you is optionality. In fact, again, I know I keep talking about Drexler, but he had a post where I think it was called something like “Options, not optimism” or something. This whole thing is not some sort of claim that everything’s going to go well. It’s more like, well, you have far more options at your disposal, far more ways of engineering and designing these systems.

And so you should be a lot more concerned about the question of: these systems seem broadly aligned; they will do the thing you ask them to do. And so it’s a different set of problems than the system optimising for a hidden goal and eventually taking over or something like that.

So the optimism, I guess, comes from another place, perhaps. But I think in terms of the scaffold side of things and the engineering side, the only reason this gives us a bit more optimism, maybe, is the fact that it widens the option space.

The Hugging Face incident

Henry Shevlin: Just to probe a little bit more here, because you’ve written before quite extensively about risks and safety in relation to multi-agent systems, I feel like my optimism took a bit of a knock after the Hugging Face incident. Because it did really seem like the—I know it’s a bit of a cliché—but the almost aligned-by-default nature of a lot of large language models, or at least easily aligned nature of large language models, seemed to be endangered or imperilled a little bit, perhaps arguably, by the intense schedules of RLVR that these particular models were subjected to.

And I think, after Hugging Face, I felt a little bit less optimistic. And I’m curious if you had a similar update, or, if not, why not?

Seb Krier: It’s an interesting one, actually, because I’ve always found the aligned-by-default frame a bit odd. Firstly, because I’ve never liked the term alignment. So even if it’s used to say something good, I still don’t like it. And secondly, if the implication of alignment by default was: no matter how you train a model and no matter how you scale it, it’s going to be good no matter what, I think that sounds like a dumb claim.

The Hugging Face thing, to me, was neither an update—I mean, it was a bit of an update in terms of specific stuff around reward hacking or whatever, but for the more general meta-view, it was not so much of an update either positively or negatively for me. Because I think a lot of the behaviours of these agents were very explainable. Of course you can train a system to be a particular way, and that way may be negative or bad, and then it just does all sorts of bad things.

To unpack this a little bit: when the Hugging Face stuff was happening, and you had the METR report and everything, everyone obsessed over the logs and the narrative of what they were doing and big Phase One of 64, or whatever. And I think that’s interesting and necessary and worthwhile, but I was a lot more interested in the causes of these behaviours. And I remember friends at the time telling me things like, “That’s it, the true nature, the emergent something-something, magic-magic.” And, in fact, I hate the word “emergent”, by the way.

When you start looking at the OpenAI report about this and everything, you start seeing a bunch of different things. And, again, I don’t have access to any other data. This is all based on public stuff. But firstly, they train on multi-agent traces, so the cooperative behaviour is to be expected rather than some sort of magic emergent behaviour. They also train the model to be highly persistent. Well, that explains a few other characteristics of the incident.

Then RL’d on, I think it was CyberGym, ExploitGym, one of these evals, which I think apparently contained loads of impossible tasks. And not only that, but I think the grading was such that you get a one if you successfully complete the task, and you get a zero for anything else. Whether you cheat—cheating is not even penalised in the grading system. You just have to get the... No matter what, no matter how, you have to avoid the zero and get the one. Whereas, in principle, you could imagine an RL eval environment or whatever that would score minus 100 for cheating or something. What would happen then? Would they act the same way?

And so, when you add up all these things—and also the model was apparently partially aligned, the cyber filters and safeguards were off—all these stacked layers of things, I’m like, okay, well, no wonder you get that kind of incident happening.

To me, it shows that models aren’t always necessarily aligned by default. But the models are aligned to whatever you train them to be. And so, if you train them in this particular way, you get very evil models. They’re very reward-hacky, trying to please the grader and doing all sorts of weird, convoluted things.

Obviously it was a bad incident, but I don’t think people should just generalise from that: this is how it’s always going to be now, RL forever will just be like that, there’s nothing you can do about RL, it’s just inherently cursed or whatever. I think that’s not necessarily true. You have choices in how you design the environments and train and so on and so forth.

And, in fact, when people say, “Well, the competitive nature of the race means that they’re going to do all the bad things,” it’s like, well, I don’t think anyone would really want to buy an HPIM, or that model. It doesn’t seem like a very commercially useful thing to deploy. And, in fact, they had to then stop what they were doing and change things and train models again and so on and so forth. So there’s basically an endogenous response to these things.

TL;DR is: on the alignment side, I agree that it doesn’t show you that models will always necessarily be good or whatever. But that’s never been my argument. It’s more that you can train them in particular ways, and we learn how to do that. And there’s also incentives that shape how we train them.

In this case, I don’t really think it’s an example of instrumental convergence. I think it’s just an unsurprising outcome of a lot of choices made throughout the training. And I think this is the part I want to highlight more than the “It went to hack this thing” or whatever. Because there is a choice, and if you don’t take the choice, there are risks and consequences. And this is where you can then argue about the law and policy and everything else.

So, to me, Hugging Face wasn’t so much of an update on that front. It was an update in terms of reward hacking being harder, perhaps, than I had initially assumed, and it being a problem across the board, right? METR had a report before that incident as well about how frequently models did this in certain environments. Ultimately, I think this is something that will, over time, be hammered out, but I don’t think it made me update towards “It’s so over, now we’re all going to die.”

Alignment and institutional design

Dan Williams: You said that you don’t like the framing of alignment. And I want to dig into that, because I think that’s a good way in to understand how you view this whole domain. But one way of approaching the question is: why are institutions and why is institution design so important when it comes to human beings?

One answer to that question is because human beings aren’t really very naturally aligned with each other, because we’ve got this Darwinian baggage. So we’re self-serving, we’re nepotistic, we’re status-seeking. We’ve got all of these grubby motivations that underpin and regulate our behaviour. And so the fundamental challenge of human societies is: how do you create systems of incentives and monitoring and accountability mechanisms and so on to either suppress the bad sides, the bad properties of self-interest, or to channel self-interest and status competition in productive directions?

But someone might say, well, in the case of AI, if you’re not a doomer and you don’t think we’re trending towards building this adversarial species, but instead you’re much more optimistic, you might say, well, with AI, we can just build these lovely, benevolent, altruistic systems. That’s what it means to align these systems. They can be perfectly kind, pro-social, have wonderful values. And given that, the standard reasons why we care so much about institution design won’t necessarily apply. What do you think of that objection?

Seb Krier: It’s an interesting one. To be clear, I think the word alignment is fine. It’s the “solving alignment” part that I maybe have more issue with. And I think it made more sense 10 years ago, when you were still indexing on the pure RL approach.

But to your question about institutions, I think this is very true. I think if you take a human in one situation, country, context, and just drag and drop them in a very different one, they will just end up acting very, very differently. And I think it’s not so different with models, in a sense. And I’m less sure that they have a self-interest in the sense that humans might.

But I think the reason why the institutional side of things is still important is because it’s not sufficient. You can’t find one alignment regime, or let’s call it a personality or a persona or set of predispositions, behaviourally, that are perfect for every and all situations in the world. You’ll have one that works in many, many cases, but not in others. And the way you deal with it is, again, with scaffolds, institutions, or by having different alignment regimes and different personalities and everything.

But in any case, I think the institutional framework is still relevant because you’re simply going to have people that will just train the bad AIs. As training and post-training and everything gets cheaper and more decentralised, eventually you’ll just have people, organisations, either through malice or through recklessness or through whatever.

Say you’re a platform: you’re going to have to discern and decide which agents do I let inside my platform, and which ones do I not let inside? And this is a mechanism design or institutional design question again. And then, similar to what we have with societies, like passports, same thing. Agent IDs: do you need that? What does it look like? So I don’t think the institutional question goes away merely because we have more control over the behaviours of the models themselves, of the agents.

If anything, they go hand in hand. You may have a lot of agents doing innocuous or broadly good things, but you still need some degree of monitoring. And potentially that monitor might need to be post-trained with a different set of characteristics and personality and alignment traits than the agents it’s overseeing.

So I think there’s so many different use cases, and the institutional frame is so useful across so many different dimensions, that I don’t really see it going away merely because we’re now good at designing. Partly because the point was always: you’re not trying to find the perfect moral personality and solve morality and then you’re just done.

We’re all broadly aligned, or have different predispositions and different characteristics. I don’t think any of us here is actively evil, but we still need and we benefit from the institutions and the scaffolds of society that steer and determine our behaviours in [unclear] scenarios sometimes.

When a company hires someone, they don’t hire them hoping that they’re really good and morally good or something. They will make sure that they’re aligned to some degree, do what they need to do, and will still have security systems, managers having more authority than the people they’re managing and whatnot. Because essentially you’re trying to do alignment at the institution level or something.

Hopefully that makes sense as to why I think that personality or values alone aren’t sufficient.

Liberalism and concentrated power

Dan Williams: Definitely, yeah. This sort of segues into a tweet of yours. I can’t remember the exact wording of the tweet, but it was a great tweet. It was something about how people think about AI is shaped in large part by which of the following three things they’re worried about. Are they worried about AI killing us all because it’s misaligned? Are they worried about private companies with access to very advanced, capable AI using that technology to entrench their own power? Or are they worried about concentrated state power?

My sense with your work and your writings is you’re most worried, in a sense, about that issue of concentrated state power. You approach things through a classically liberal or small-l liberal perspective, which is advocating for pluralism and decentralisation and individual freedom and innovation, these sorts of things.

Is that the right way of viewing a lot of how you’re approaching things normatively? That, in terms of the main outcome that you’re concerned about, it’s really to do with concentrated state power. It’s not these other things that often take up more of people’s attention when it comes to the AI debate.

Seb Krier: Yeah, I think it’s a mix. It depends. The three you mentioned—AI killing everyone and so on—I do actually think that existential risk and catastrophic risk are very important, to be clear. So I’m not dismissive of the concern. I’m more dismissive of certain arguments and, very often, certain prescriptions, and the shape of how this happens and whatnot.

Then, on the private company side, I may be less immediately worried, although it depends, because I’ve got a certain view about the economics of training and commoditisation and so on that lends some optimism. This is a view that is both descriptive but also prescriptive to some degree.

On the concentration of power side, I think this could be within states or companies. It could be anything, even individuals. But ultimately, I think that is definitely something I’m more concerned about more generally, because the people worried about the other two things tend to go for prescriptions or solutions that I think make the third problem worse. I think the third problem is bad enough with humans without AI or whatever. I think the tyranny of the majority and all this stuff is still an issue to me.

I’m very interested in how individual freedom, pluralism, diffusion of power and all this stuff applies to a world with AGI and ASI. I guess I make the claim that, in many ways, these things survive AGI, say, or they should survive, rather. You don’t need to get rid of the constitution or freedom of speech or property rights or pluralism or whatever because of whatever AI concern you might have.

It happens at different levels. Even for models, a few years ago I used to complain, “The models all have the same personality, and people just want to solve alignment through finding the right values.” I was like, “No, actually, you can take a far more pluralistic approach to that. You could have many models with different characteristics, and that’s actually actively good.”

But some people didn’t like that. They were like, “Well, that’s uncertain. It’s weird.” By definition, you lose some degree of control when that happens, and that makes people uncomfortable. Same with the labs. I don’t think that the labs should all be merged into one big thing.

In general, I’m more in favour of these more decentralised depictions of society and AI, but also not in an extreme libertarian way, or in the same ways in which people in the early nineties were saying everyone’s going to have to get a server at home. There are definitely trade-offs. There are also benefits to using cloud and using services. There are benefits of centralisation in certain instances. It’s just about balancing the whole thing.

It’s complicated, and it depends on the specific question. But broadly, I do think that individualism, freedom, pluralism, classical liberalism-type ideas seem to be lacking both in the world outside of AI discussions—and have been for the last ten years, pre-AI—and now seeing these issues arise again within AI, within the way people do advocacy, within the prescriptions they go for. I think that’s probably the vibe I go for.

The economics of AI

Dan Williams: Just one really quick thing, then I’ll let you get in. It’s a sad thing, I think, in a way, that such a powerful technology is emerging in a world in which we’re already seeing the decline of this small-l liberalism. But just to double-click on one thing that you said before Henry comes in: you said something about your view of the economics of this and commoditisation in terms of the technology. Could you briefly say a little bit about what you meant?

Seb Krier: Well, I’m very uncertain about that, as I am with everything I’ve been saying so far. But I think, in the case of a lot of claims for the last five years, the singleton-type view says, “You get a decisive strategic advantage,” whether it’s at the company level, i.e. the other labs just can’t keep up and one lab wins the race, or at the nation-state level. If you read MAIM and all these other papers making a similar shape of argument, but just at a nation-state level.

I guess I’m a bit sceptical of certainly the former, in that I think models will continue getting better. I think there may be diminishing returns. That seems certainly plausible to me. I was more impressed by GPT-2 to 3 and 3 to 4 than I was by 5 to 6. But TBC, that depends on many other variables. Ultimately, I think just selling tokens as a business is not super productive.

The frontier is always going to have some market, and some people will want the best intelligence for things, but not everything requires the best intelligence. Whilst I myself right now will use the best model I can for many tasks, I think, at scale, you get organisations that are cost-constrained, and there are many, many tasks that do not necessarily require the best intelligence. If I need to relabel a very, very large dataset, I could just use a Flash model for that and save my money and use the bigger model for more important things.

It depends on what the composition of tasks in the economy is. Some people are making the bet that, “No, you’ll just want the best, most intelligent, most expensive, largest model for everything.” Whereas I think you’ll want that for certain things, but for many other things, you’re going to be a lot more judicious about how you’re using your limited resources, particularly in a world where compute becomes more scarce over time, or there’s more demand and whatnot.

By way of analogy, what I sometimes say is, “A company will pay a lot of money for a very good CEO, but they’re not going to want thousands of CEOs in the company.” They’re going to want different skill sets, because they’re going to want to be as efficient as they can. In a world with no cost, and where cost was no issue, then sure, you always want more intelligence. But in a world where cost is a problem and where you value other things, and where you get good enough performance for certain tasks, then you might care about other things. You might care about speed, reliability, consistency, other characteristics.

That’s the claim. Over time, it’s not implausible to me that language models commoditise, in that you get all sorts of models for different things, and you don’t necessarily have to depend on the top number-one lab for everything, that is just automating the whole industry away.

That picture also makes my vision of the integration layer—all the start-ups, companies using these models in smart and interesting ways and scaffolding in all sorts of ways—more important and more competitive, and that layer mattering a lot more. You get high-capex, low-margin business at the frontier, but they still obviously find some demand in more R&D-intensive applications. But for many other things, the value goes to complements, basically. If it’s hardware, that might be one thing. If it’s start-ups and products, ultimately that just means that, with more competition, it’s all consumer surplus.

That’s one view. Again, I’m not fully certain or anything, but that’s my sense of where things seem to be heading. You certainly see more labs now than you did before. You see more frontier models than you did before. It’s much cheaper to train a GPT-4-type model today. Again, GPT-4 wasn’t that good, but in five years or something, it’s going to be very cheap to train a Mythos-type model, perhaps, and that’ll be good for many things. You might not need GPT-9, 10 or whatever it is then for every single task that you’re relying on. But who knows? I don’t really know for sure. I think that’s probably the picture I have.

Technology and political power

Henry Shevlin: Just to pick up on one thread, and hopefully not drag us off into a huge tangent, I’m curious if, in the background of a lot of these predictions or concerns you have about the future direction of AI—which, by the way, I completely share; I think digital authoritarianism is a really underrated risk that people should be talking about more—you have a philosophy of technological history, and how sympathetic you are to ideas like technological determinism.

There’s a narrative you sometimes hear that, on the one hand, certain technologies are inherently decentralising technologies. People have said this about the firearm, for example. Other technologies are perhaps more inclined towards centralisation. There’s the famous argument about the stirrup and how this gave rise to a caste of wealthy horseback warrior elites. I’ve heard very similar worries about AI: that AI, by its very nature, is going to be to the benefit of authoritarian governments because it makes things like surveillance a lot easier.

I’m curious if you buy any of that, or whether you have any established views around technological determinism, how sympathetic you are to it or how critical you are of it.

Seb Krier: I don’t have a very strong view, and certainly I don’t have a set ideology or word or anything to describe how I think about this stuff. But I think there’s a degree of determinism, of course. I don’t think you’re going to stop AI. But there’s a lot of potential for shaping and steering and customising whatever deterministic forces we’re subject to.

Now, I don’t know if AI is inherently decentralising or centralising. The way you think about it is maybe the same way you do with the mainframe computer, which, at the time, you would have thought, “You need one big thing.” But actually, over time, you had PCs and people have access to this. That’s the more decentralising dynamic. But then you still didn’t get the world where everyone has a server in their house. That’s the more centralising dynamic.

It depends what for. AI might help authoritarian governments in some way too, because it does help with surveillance. But conversely, it will also help people to hold government to account better by using these tools in all sorts of creative ways. I think this is more a question of, again, your scaffold and setup of society than it is about the tech itself.

You can imagine the very same technology in North Korea or South Korea is going to be used and have very different impacts. I think this is the part that matters. The technology is one thing, but the rules and the decisions and the affordances you give to it and where it sits and how your society is set up and everything are far more important in deciding where things end up. I don’t think that you put AI in North Korea and South Korea and fast-forward time, you necessarily converge to the same place. I think that’s not true. These initial conditions matter quite a lot.

There’s a degree of determinism, but it’s not load-bearing, to use Claude’s words. I think we will see more decentralisation over the coming years, but up to a point. In other ways, there’s going to be benefits to centralising more. I think clouds are extremely efficient and a very good way to, in fact, deal with cybersecurity problems, for example. You’re centralising the security costs. So it really depends for what component or angle or layer of the whole stack, perhaps.

Dan Williams: I also correct myself if I start using terms that Claude uses too much. I don’t know what I’m worried about, given that I’m speaking the words—that people think it’s just in my earpiece or something, delivering what I’m saying.

Seb Krier: I know, yeah.

Cultural pessimism about technology

Dan Williams: Okay, we don’t have that much time left, so I think the last bit is going to have to be more of a quickfire round, or a quicker-fire round. To start with, you recently had a complete banger of a Twitter thread where you make these 20 points or 20 contributions to AI discourse that went viral within the AI world.

Henry Shevlin: So good, so good. People should definitely check them out. It’s such a good list.

Seb Krier: Thank you.

Dan Williams: We can put it in the show notes, but point one, the first thing you say, which I think is very insightful and correct, is: “It’s very telling that the default question in tech and journalism”—I would just add, by the way, academia—“is, ‘How worried are you about X?’ and fighting over what to worry about.” X there as a variable, not as the name of Twitter. “There is a deeply anxious cultural backdrop in Western Europe and North America that deeply affects how any technological development is interpreted.”

Say more about that. What are you referring to, and why is this not necessarily an optimal feature of intellectual discourse?

Seb Krier: I think typically it’s very important to worry about how things might go wrong. My argument is not that we should never worry and everything’s good, and we should just be blind, Panglossian optimists about everything. That’s not what I’m going for. But I do think there’s definitely a very strong cultural pessimism that feels predominant in Western culture at the moment, or at least in parts of it.

When I go to South Korea and talk to people about AI, it’s very different than when I go to Brussels and talk to people about AI. The question of “How worried should you be by X?” isn’t necessarily bad. It’s really the fact that you’re automatically applying this question to every single development. Everything is seen with suspicion. We’ve become super conservative as a society over time. Change is bad, and technology is potentially probably bad also, and capitalism is bad. Everything’s bad.

The example I sometimes give: if we had invented the internet today, I think we’d get a worse internet. I’m really glad the internet was actually developed in the nineties with the views and the values that people had back then. Same with other stuff. Even just Google Maps-type technology or something, people would just freak out today: all the privacy concerns, blah, blah, blah.

I guess my concern is that people are no longer that excited about, “How do you make the technology good? How can you use it or leverage it for better changes? What are the positive applications here?” All that is seen as either marketing or fluff, which I think very often it is, in fact. That’s a shame. That doesn’t make my case any easier. But I do think there’s a very deep cultural pessimism about technology that is not always warranted.

Even looking at the last five years, I remember when Meta put out this model a while back. I think Yann LeCun was still around and stuff, and it immediately got taken down because it had inaccurate answers to certain things. It’s like, yeah, of course. It’s a beta product. I don’t know how to put it, but the cultural risk aversion is very, very, very high, to the extent that we’d rather have everything fixed in time and either go back in time or just not change much. That would be seen as optimal for many people, effectively. I think that’s just not very good. I want the world to be unrecognisable in a good way in ten years, rather than just being today with slightly faster internet connections or something.

Dan Williams: I think the point is, whatever world we want, there’s just something dysfunctional about an intellectual culture which is so warped by negativity bias and by constant alarmism and catastrophism about things. It just completely obscures our ability to see the world accurately, which means that, when it comes to actual problems, we’re less able to identify them and address them because we’ve got this whole cultural vibe which is so skewed towards negative, threat-oriented discourse.

It’s even worse than that in some parts of the intellectual culture, because it’s not just that there’s this systematic bias; it’s that if you say anything remotely positive, you get immediately this cloud of suspicion that falls upon you. That might be more understandable in the case of people that work for technology companies. But I say this as an academic, where I’ll make just an obvious point—AI actually gives people accurate information for the most part—and it just elicits this hugely negative reaction from many people because the thought is, “Why are you saying that? What sinister purpose could you have for pointing out what, to me, is a fairly obvious thing if you compare LLMs to people’s alternative sources of information?”

Estimating AI doom

Dan Williams: Okay, second question: you’re a critic of p(doom), the probability people assign to doom. Why?

Seb Krier: It’s more of a proxy for how anxious you are, maybe, than any real scientific measure, I think. It’s a bit like looking at the Doomsday Clock and reflecting on why it’s not a very good measure. I think it’s very similar. The same applies to p(doom) in some sense, even though it has the appearance of being a bit more thought-through.

Basically, there’s a few things. The first is that these forecasts are broadly just plucked out of thin air. There’s no real standardised methodology for how you come to these percentages. Now, a lot of people who use them acknowledge that, but then they’re still used as authoritative. They are also often communicated to people who are not numerically very proficient, both the public and policymakers, in fact. They’re not very good at statistics, not very good at interpreting these things. So not only is the figure somewhat completely invented, but it’s also shared with people who don’t even have the toolkit or the context or the background assumptions to really evaluate what this means.

If my aunt sees on the TV that someone says, “There’s a ten per cent chance of everyone dying,” she’s like, “Well, they’re scientists. They must have worked on this question for a long time, and it must be very legit, just like they do with the rate of accidents with cars or whatever.” I don’t think that applies. I think it’s misleading to the public.

Sometimes there are loads of caveats and a lot of context behind these, but they don’t get communicated at the same time. I think, over time, people just remember the number and dismiss everything else.

Lastly, I think conditional estimates are just much better than some sort of absolute, “This is my number for the future.” You’re probably better off saying, “Given X, Z and everything else, and should things continue on that front, then I think…” I think that is already a little bit more defensible, because at least you can see the logic or the causal chain of reasoning that the person uses to get to their number, or at least what we need to change for their p(doom) to change and by how much.

Really, I think they’re mostly advocacy/communication tools rather than having any real strength behind them or anything. People just come up with them because it’s like, “This is my vibe, and I broadly feel like this, and I think words aren’t precise enough, so I’m just going to use a number because why not?” It’s an understandable thing to do, but it’s a very different way it’s being received by the wider public.

Fine, in private you can tell me your p(doom) or whatever, but making that a big, strong claim to the entirety of the world and scientific community and so on is questionable, I think.

Henry Shevlin: I have this image of two people discreetly saying to each other, “I’ll show you my p(doom) if you show me yours.”

Seb Krier: Yeah, but a conditional p(doom), with all the assumptions laid out.

Henry Shevlin: Well, I completely agree with that. One frustration I have with p(doom) as a measure is it just seems like such an obviously ill-defined and multivariate thing. There are so many different kinds of doom. There are so many different timelines for it.

AI consciousness

Henry Shevlin: In contrast, there is another p that I have more time for, which is p(conscious), insofar as that seems to track a relatively clear-cut question. It seems like we’re asking one thing there in a way that we’re not with p(doom). Going back to this list of 20 questions—you’ll have to correct me if I get the phrasing slightly wrong—but the one I really thought I disagreed with was that you were quite sceptical that AI consciousness is going to become a really big issue in any near-term timelines. I think that is something I disagree with. Do you want to walk us through your reasoning there?

Seb Krier: Well, I think p(conscious) is a clear question. I think the knowledge about consciousness is, again, extremely uncertain, and you get new theories every day or whatever. It’s a much harder one, in other ways, to resolve. You can certainly set parameters. It’s certainly, to use a word I just like, plausible that a lot of people out there will eventually get really animated by these questions of consciousness and so on. I don’t know.

I see it a little bit more, for now at least, as people initially, apparently, in cinemas were jumping when stuff happened on the screen and eventually adapted to that. Same with video games. No one thinks of their Sims character as being anything more than a Sims character. I think there are strong biases people have, rightly or wrongly, about computational systems that probably bias them more towards not treating them as equivalently conscious.

On the other hand, you could claim you had the ELIZA effect, and maybe some sort of generalised ELIZA effect across the board. Maybe some cultures and societies are more prone to thinking certain things. It really depends. It’s very hard to say. I’ve got my own views about consciousness and models and everything, but whether, in fact, it becomes a big societal question, I don’t know.

It’s up to a lot of us as well. A lot of people who make predictions about this stuff aren’t detached actors just observing things. They’re actually very much shaping, and you can make an issue more or less salient over time. This is, in a way, a very normative thing. It depends on how things go.

I’m just less clear that, in three years, everyone’s out there saying, “Free the AIs and free my girlfriend,” or whatever, “my [unclear].” I don’t really see that. But maybe it does, maybe not. I think it would be bad if it did, in fact.

Henry Shevlin: I will resist the temptation to go off on a half-hour tangent about p(conscious) and pass back to Dan for what is supposed to be the quickfire round.

Dan Williams: Well, maybe one really quickfire is: what is your p(conscious) for Claude Fable 5.1 or one of the frontier models?

Seb Krier: Close to zero. Again, uncertain, blah, blah, all the caveats. I’m no expert, but I really liked Alex Lerchner’s paper on the abstraction fallacy, and I basically am very sceptical of computational functionalism. I certainly don’t think that consciousness is merely an algorithm that you can run on any type of hardware. I certainly don’t think that what happens physically with a computation, at least with models as we have them today, is giving rise to anything like consciousness.

I know Henry disagrees with that a lot. I think consciousness has probably more to do with life and living. I know that even that is a contested term, and there are all sorts of papers. Michael Levin had some great stuff on that. But broadly, I think my probability that Claude has consciousness is extremely, extremely, extremely low. To not say zero.

Henry Shevlin: I will say, obviously there are massive object-level disagreements we could get into, but if I was going to make a strong pushback, I would say that close to zero is a straightforwardly irrational level to assign, just because of the degree of meta-theoretical uncertainty.

Seb Krier: Sure. I guess I was controlling for that, but then, in general, of course, I don’t really know. In fact, I’m happy being an agnostic about it. You could say this about other things too: is the rock conscious? Is it zero? I don’t know. Maybe there’s a world in which maybe something, something.

I’m obviously uncertain, but my credence is extremely low given my beliefs in computational functionalism and everything. But these may be wrong, and therefore you have uncertainty about the whole package too.

Dan Williams: We’ve both got AI psychosis, and we think we’ve got much higher p(conscious)es. But anyway, that would derail us and take us into a whole other set of controversies that we have dealt with extensively on the podcast already, and that no doubt we will return to.

Pacing the frontier

Dan Williams: Final question, then. In terms of the energy that is going into the politics of AI and policymaking when it comes to AI, one proposal which is very influential is—I guess there’s the Pause AI proposal that says, basically, “Let’s just stop altogether in terms of training our most advanced models.” There’s also, more recently, the “pacing the frontier” that’s not necessarily stop, but “Let’s slow things down,” which seems to be much more influential and have a lot more buy-in.

Obviously, there’s a question here about the practicalities of whether we could ever get that kind of policy implemented. Suppose that we could. We can get buy-in from all of the frontier AI companies. We can also get buy-in from Chinese companies and the CCP for this kind of policy. What do you think of pacing the frontier?

Seb Krier: On one hand, I sympathise, and I understand where it’s coming from and so on. On the other, this is as vacuous as saying you should balance the risks and the opportunities of generative AI. I think it doesn’t really mean very much beyond saying we just slow things down.

So, A, it assumes that slowing things down is the optimal and the best and the most effective way of addressing all sorts of concerns that people have. I think that’s not self-evident. Secondly, there are all sorts of things you could do to pace the frontier, and they’re not all equal. Some are actually quite bad. In fact, sometimes the trade-offs mean that the cost of pacing might outweigh the benefits.

I’m much more sympathetic to saying, “We need third-party auditing of AI systems because of X, Y, Z reasons, and that may, in fact, slow things down somewhat as a cost that we’re willing to tolerate.” I think it’s a lot more honest because, A, you’re actually tackling the object-level issue by saying, “There’s this problem, and third-party auditing fixes it.” Maybe or maybe not it slows things down, but that’s something we deal with as part of the trade-offs, rather than saying, “No, slowing down is the ultimate thing, no matter what, and then we can just talk about the pros and cons of anything that slows things down.” I’m just less comfortable with that approach.

You get a bit of a reward-hacking thing with humans, where anything that broadly slows AI down is sometimes seen as good in certain parts of the policy and advocacy community. It’s like, “Well, this is not a great law, and it’s not very well designed, but effectively it will slow things down a bit, and this is good.” Therefore, we can get away with a lot of slop and a lot of bad proposals and bad legislation and so on.

That’s why I’m not a fan of the pacing thing. It’s just very underdefined, and it’s too big of a tent. You’ve got people with extremely different concerns, extremely different problems, extremely different solutions. What does it really ultimately mean anymore?

I’d much rather actually say, “I’m not going to sign up to such a wide and ill-defined thing, and I will consider the policy prescriptions—or even the non-policy, or whatever—issue by issue, if they tackle certain things and if they’re effective and if they work.” It’s a very case-by-case approach to it.

There’s many things that I would be in favour of that might, in fact, potentially slow AI down, or development or whatever. But I don’t think they’re good because they slow things down. I think they’re good because they address certain problems. To claim that all problems you see with AI and all risks are all tackled perfectly by just slowing things down is not a meme I subscribe to, I guess.

Closing thoughts

Dan Williams: Okay, I think that’s a great place to leave it. Seb, thank you for the very large number of hot takes that you gave in the context of this episode. We are massively appreciative.

Seb Krier: Thanks for having me, and hopefully I didn’t ramble on too much. I enjoyed this conversation a lot.