Is AI a technology, maybe even a normal technology, or is it something radically different?
In this episode, Henry Shevlin and I discuss this question with Professor Arvind Narayanan of Princeton University. Arvind is one of the most important voices writing about AI today. With Sayash Kapoor, he authored the highly influential essay “AI as Normal Technology”. He has also just co-founded STABL, a new research organisation on how society should adapt to advanced AI.
We talk about how to understand Arvind’s worldview and which popular ideas about AI it is opposed to. Along the way, we also touch on some other topics, including consciousness, AI’s impact on the information environment, and the surreal experience of using AI today.
We discuss:
Why Arvind thinks the history of electricity can teach us about AI
The gap between what AI can do and how reliably it does it
Why he doubts that recursive self-improvement will produce superintelligence in a lab
Whether superintelligence is a coherent concept, or a choice about how much power we give to AI systems
Whether we can avoid building conscious AI, and whether conscious AI would matter morally
The three kinds of AI persuasion, and which of them worries Arvind
Why the AI misinformation epidemic never arrived, and why AI companions might be a bigger problem
Why so many confident predictions about AI have been wrong
Why robots might displace blue-collar jobs faster than AI displaces white-collar ones
Whether a small chance that AI 2027 is right should drive public policy
The surreal experience of working with AI, and whether it will soon write better books than we can
Links and further reading
AI as Normal Technology — Arvind Narayanan and Sayash Kapoor’s 2025 essay for the Knight First Amendment Institute, which sets out the worldview we discuss.
The missing quadrant in responses to AI — the essay announcing STABL, by Sayash Kapoor, Rishi Bommasani, and Arvind.
Towards a Science of AI Agent Reliability — the paper led by Stephan Rabanser that measures the reliability of AI agents separately from their capability.
Can AI agents conduct open-ended AI research? — Peter Kirgis, Sayash Kapoor, Arvind, and colleagues test whether AI agents can do AI research when the question is open-ended.
What will be left for us to work on? — Arvind’s ICML 2026 keynote, including his reflections on adapting to AI.
We Looked at 78 Election Deepfakes. Political Misinformation is not an AI Problem. — Sayash Kapoor and Arvind on AI and misinformation in the 2024 elections.
AI existential risk probabilities are too unreliable to inform policy — Arvind and Sayash Kapoor on forecasts of AI catastrophe and public policy.
AI 2027 — the scenario from Daniel Kokotajlo and colleagues at the AI Futures Project, in which the automation of AI research leads rapidly to superintelligence.
AI As Profoundly Abnormal Technology — Scott Alexander’s response to “AI as Normal Technology” for the AI Futures Project.
How Close Is Recursive Self-Improvement? — the Forecasting Research Institute report that Arvind mentions.
Durably reducing conspiracy beliefs through dialogues with AI — Thomas Costello, Gordon Pennycook, and David Rand’s study in Science of the first kind of AI persuasion we discuss.
How AI Will Reshape Public Opinion — my essay on why large language models are a kind of anti-social media.
AI-based disinformation is probably not a major threat to democracy — my 2024 essay on why fears about AI and disinformation are exaggerated.
AI Sessions #6: AI Companions and Consciousness — our earlier conversation with Rose Guingrich about social AI.
Transcript
Please note that this transcript was lightly AI-edited and may contain minor mistakes.
Introduction
Dan Williams: Welcome back to the Conspicuous Cognition podcast. I’m Dan Williams, back with my co-host Henry Shevlin. Is AI a technology, maybe even a normal technology, or is it something radically different? That’s what we’ll be discussing with our guest, Professor Arvind Narayanan of Princeton University. Arvind is one of the most important voices writing about AI today.
With Sayash Kapoor, he authored the highly influential essay “AI as Normal Technology”. He has also just co-founded STABL, a new research organisation on how society should adapt to advanced AI. We’re going to talk about how to understand Arvind’s worldview and which popular ideas about AI it is opposed to. Along the way, we’ll also touch on some other topics, including consciousness, AI’s impact on the information environment, and the surreal experience of using AI today. Arvind, welcome to the podcast.
Arvind Narayanan: Hi, Dan and Henry. Thank you so much for having me. This is such a treat for me. I’ve really enjoyed both of your writing. Dan’s writing has had a profound impact on my own work and thinking. And Henry, we share an advisee: Rose Guingrich, whose PhD committee you were on, is now doing a postdoc with me. So, very grateful for all of that.
Henry Shevlin: And also to plug that we had Rose Guingrich as one of our first guests on the show. So if anyone wants to check out her discussion on social AI, go back and listen to that episode.
What Past Technologies Can Teach Us About AI
Dan: And “AI as Normal Technology” is probably one of the topics we’ve covered most across such a large number of the episodes on the podcast. I think there’s one classic objection, not just to the title “AI as Normal Technology” but to the idea that we can draw lessons from previous general-purpose technologies to understand some things about AI and its impact on the world, most fundamentally the idea that these impacts are likely to take time, that there are these slow processes of diffusion.
The objection is: AI, these systems that we’re building, they’re thinking machines. They’re machines with minds that can understand, that can plan, deliberate, forecast, pursue actions as a way of achieving their goals, coordinate with each other, and so on. And that’s simply nothing like any technology we’ve built before. What do you say to that general objection?
Arvind: Yeah, that’s a great question. I’ll say a few things. One, the lessons that we draw are less about the technology and more about the bottlenecks, about the organisations, the human processes that ultimately are going to be adopting AI or any real technology. And I think those lessons are very valuable.
When we look at early attempts to use electricity in factories, they tried to use it as kind of a drop-in replacement for these big steam boilers. And I think that term is very helpful. We hear a lot about how AI is going to be a drop-in replacement for remote workers. Well, that didn’t work. It took 40 years to really figure out that the way to really take advantage of electricity was to consider that it was a much more portable technology, that you could move it and generate it where you needed it, and rethink the assembly line, the division of labour between people, lots of other things.
And so to me, those past experiences serve as kind of hypotheses that we can use to then analyse AI. It’s not “that happened before, therefore that has to happen now”. I do use these analogies and metaphors a lot to give people an intuition, but when you look at the essay, a lot of the argument is specific to AI, and a lot of our research is about measuring some of those bottlenecks that are specific to AI.
In fact, this way of thinking based on history got us to realise, and this is work led by Stephan Rabanser, who’s a postdoc working with me, that too many of these predictions about rapid AI impacts are made based on these capability benchmarks. But then when you look at what those organisational bottlenecks are, they often end up being about the reliability of AI systems, which forces humans, for the time being and I think for a considerable time in the future, to still be in control. And that really limits the ways in which AI is getting adopted.
And we realised that nobody had measured that, as far as we could tell. So this paper that we put out earlier this year was the first comprehensive attempt to measure the reliability of AI as a separate property from capability. So, looking at not just whether AI can do a task once, but whether it can do it in a manner that is predictable, figure out its own mistakes, recover from them, various other things, looking at other domains where safety is critical, like nuclear and aviation and so forth, and drawing lessons from there. And what we found is that over most of the last 24 months, a period during which capability on benchmarks has been increasing very rapidly, reliability only increased by five or ten percentage points.
So that is one example of the specific kind of arguments that we make about AI’s bottlenecks, and that work is still ongoing. We’re launching, as part of STABL, the new organisation that you mentioned, a whole research agenda on mapping these bottlenecks specific to AI. So that’s where a lot of the specifics of the argument come from, and the history is more for inspiration and hypothesising and explanation.
Henry: Just to probe one part of this parallel with electricity, which I think is really useful and I’ve used myself quite a lot: I think one thing you could point to there is that electricity relied on a large amount of physical capital. Factories had to be rebuilt on multi-level floors to take full advantage of electricity. Whereas the underlying physical capital around AI, basically data centres, seems much more flexible. Everyone can already plug in to the latest model as soon as it’s released. We don’t need to buy a whole new set of servers for our office or home just because the new model is updated.
That suggests at least some of the kind of bottlenecks in distribution and physical capital that apply to electricity wouldn’t apply to AI. I’m just curious how much you think that changes questions around bottlenecks and diffusion.
Arvind: Some of the bottlenecks, yeah. And again, that’s why we need this new research agenda of mapping what the bottlenecks for AI are. I think it’s also important to keep in mind that as we look at more and more technologically advanced societies, in many ways there are more bottlenecks, because AI is entering a world that is already highly automated in many ways. A lot of the low-hanging fruit has been picked by earlier waves of automation based on more traditional code and computing technology.
And when you look at the specific areas where generative AI and these new agentic types of AI based on large language models shine, what they specifically shine at is a lot of tasks involving more judgement, for instance, than traditional code was capable of. But that also introduces these new kinds of bottlenecks that I’ve been talking about. It can do the task, it is capable, but it’s not reliable.
So that’s a kind of bottleneck we didn’t have to worry about with traditional automation. If you can automate a thing once, by the very nature of automation, it can do it every time. So this is a profoundly new concept, to automate things in a way that is so unreliable, that we haven’t had to deal with with previous technologies. So yes, some old bottlenecks go away, but new ones come into play.
Capability, Reliability, and Jaggedness
Dan: So I think this distinction between capability and reliability is such a valuable part of the work that you’ve done. But I think a counter is going to be: to the extent that there is this capability-reliability gap at the moment, ultimately that’s because the capabilities just haven’t come along far enough. And if you look at the trend when it comes to AI progress, yes, at the moment there are these issues with kind of jagged, brittle intelligence lacking certain kinds of capabilities that human beings have.
So it’s true that if you look at the world now and over the past few years, it looks like it’s a world where you’ve got all of these complex issues to do with the slowness of diffusion and bottlenecks and so on. But as these systems get much more advanced, as they get more capable, these kinds of issues just aren’t going to arise. What do you think of that kind of objection?
Arvind: For sure, capabilities are improving. We’re not capability sceptics, and we’ve tried to be clear about that in all of our writing. We do think there is a difference between verifiable tasks, such as coding, such as cybersecurity, and less verifiable or unverifiable tasks like writing. When you look at the trajectory of the quality of AI writing over the last few years and compare that to coding or math or whatever, it’s very, very different, and it’s unclear how much improvement there has been in a lot of unverifiable tasks. So that’s one reason for our scepticism.
The way we see it, the jaggedness is actually not closing. Things are getting more jagged. And in many ways the jaggedness is accelerating, because once a task is verifiable, once you have perhaps even a very expensive kind of AI system that can perform the task, that can then be used as a reference to train new and more efficient AI systems that can then be diffused, perhaps, at a lower cost more rapidly throughout the economy. So there are many reasons to think that jaggedness is increasing. And certainly, when you look over a 20- or 30-year timescale, perhaps even less, maybe 10 years, we do think a lot of these reliability issues would go away.
Perhaps one crux of disagreement that we haven’t gotten to yet is about whether there will be some moment of discontinuity. That is maybe the fundamental thing we disagree on. We’re not sceptical that over the long run this is going to have impacts as profound as the Industrial Revolution. A very rough way, perhaps a little oversimplified, that I think about it is that what the Industrial Revolution did for physical labour, AI is going to do for most types of cognitive work.
But I think that’s going to happen in a continuous and not discontinuous way, giving us many levers and time for adaptation. It’s not going to be this one moment where recursive self-improvement, for instance, leads to superintelligence and immediate replacement of people across the economy, uncontrollable safety risks, etc. So those are the aspects that we push back on. We’re not pushing back on the idea that it’s going to get more useful gradually.
Recursive Self-Improvement and Its Limits
Dan: I think it would be helpful to dig a little bit more there into recursive self-improvement and superintelligence, because you’ve got a few different sources of scepticism there, as I understand it. One concerns the process of recursive self-improvement itself, and the other concerns what it’s leading to. And I suspect on that latter issue of what it’s leading to, your critics would say in some sense you are a kind of capability sceptic, because you’re sceptical of the kinds of capabilities that they’re imagining on the other end of this.
But if we take recursive self-improvement first: you said there’s this distinction between these verifiable domains like maths and coding and these other more messy, open-ended domains, which are relevant to lots of real-world activity and also potentially things like writing as well. But I think the view from the rationalist, effective altruist, AI 2027 worldview is actually that when it comes to recursive self-improvement, when it comes to AI systems building better and better AI systems, really that’s to do with verifiable domains, the kinds of domains that even current AI can get really, really good at.
And if it gets good enough and it can automate the process of designing better AI systems, then you’ll be able to design completely different kinds of capabilities that are much better at generalising from experience and acquiring these more open-ended capabilities. So as I understand it, that’s the core intelligence explosion, recursive self-improvement story. Could you say a little bit more about what specific aspects of that story you’re most sceptical of?
Arvind: I think there’s a big leap of logic there. Yes, it might be possible to automate what today a lot of human researchers are doing in the process of developing AI. I do already have some scepticism there about how close that is. One kind of graph that people look at is how much of the AI research tasks are being done by AI itself, and one might expect that once that hits 100%, then you’re at the threshold of something that you can call recursive self-improvement. I don’t think that’s the case.
When we saw that with coding, a lot of people thought that once AI does 100% of the coding, there’s not going to be any need for human software engineers. Well, we’ve hit that mark at many companies and there hasn’t even been a perceptible drop in the number of software engineers. Not only has it not gone to zero, it doesn’t seem to have decreased at all.
So anything that is precisely specifiable as a task, we might be comfortable delegating to AI. But even when a lot of that gets taken over, just the choice of what to do next, that becomes a greater and greater proportion of what one is thinking about. And yeah, it is possible that companies will try to delegate more and more of that as well. I do think that is why we need transparency around what is going on with respect to recursive self-improvement. I’m cautiously supportive of bans on something that gets too close to full automation of the whole process.
But let’s set all that aside. Even if it is one day the case that something close to full recursive self-improvement is possible, Sayash and I, and many others, are very sceptical of what the endpoint of that process is. Even if you can somehow speed up this computational process infinitely, we don’t think superintelligence is the kind of thing that can be built in a lab, in principle. We think a lot of the bottlenecks to the kind of superhuman capability that people imagine are external to the lab.
Some of the kinds of knowledge that we prize the most, whether it’s medical research or knowledge of how to govern a complex economy, a lot of that comes from real-world experience, from doing randomised controlled trials or observing the world gradually over a long period of time and building up that knowledge. It’s not something you can compute your way to. And ultimately, AI’s capability and intelligence are going to be bottlenecked by the same things that have limited the rate of human progress so far. So there’s a lot more to say there, but those are the main reasons for scepticism.
Henry: So just to double-click on this idea that there are going to be some domains that you can’t just train your way, in isolation, to success in: what do you think about the role of simulation here? Because there are lots of domains, economics, materials science, where, even though of course you get a lot of value from doing real-world experiments, you can get at least some of that same value through ever more intricate and complex simulations, and maybe increase the informational value of the actual real-world experiments that you do. To what extent can that reduce the bottlenecks in those domains?
Arvind: I think it can reduce it. I don’t think it can eliminate it. When we look at self-driving cars, for instance, simulations have already been an important part of how they have improved. But when you think about edge cases, the fact that you have to drive a million miles before you see a single instance of an edge case, that turns out to be really important for the safety of self-driving cars. And if you just don’t have that in your data set, you’re going to end up deploying these cars on a large scale, and then maybe a few months later you realise that you’ve hit this edge case that then leads to a big public pushback.
And it seems to me, fairly or unfairly, that we hold these automated systems to a significantly greater standard of safety than we do humans at the wheel, at least in the case of self-driving cars. And I think that’s probably going to be the case in many other highly consequential domains, whether it’s medicine or government or whatever it is.
So I think simulation has a role to play, and I’m also supportive of regulatory reforms that make it easier to get evidence from simulation. There are, in fact, regulatory barriers there that limit the extent to which you can get away with simulation in clinical testing, for instance. I don’t think there’s much of a role today because of how regulatory processes are set up. I do think it’s helpful to reform them, but you’re not going to be able to change them to such a degree that you just completely skip the clinical testing process and start treating millions of people based on lab evidence alone.
Is Superintelligence a Coherent Concept?
Henry: And just one other very quick clarification. When we had Steven Pinker on the show, I think he seemed uncomfortable or dubious just about the very concept of superintelligence. And just to be clear, you’re not coming at this from a conceptual worry that intelligence is just a poorly defined concept or superintelligence doesn’t make sense. It is just an empirical claim about our ability to get to it. So it’s a claim about capabilities, speed of growth, rather than a conceptual one. Is that right?
Arvind: I have a lot of concerns about how conceptually coherent this concept is. I just didn’t bring them up because I have perhaps even more fundamental concerns on the empirical side. But let me give you this example. In many ways I think superintelligence is a choice. Whether to label something, whether to treat something, as superintelligence is a choice.
If you go back a decade, let’s say, and a person popped up with the coding capabilities that today’s AI systems have, we would in every way have considered them the best coder history has ever seen. We would have thought of them as superhuman and we would have put them in charge of things.
These abilities, like massively writing a large amount of code, with some limitations, like not worrying too much about how easy it is for mere mortals to read and maintain and improve over time: there have been many coders like that. Again, not to the same extent as AI, but being able to be much more productive than a typical human coder. And the practice has been actually to put them in charge and to put others who are perhaps less capable in charge of the cleanup, so to speak.
So we could choose to treat today’s coding agents like that, with human programmers relegated to the role of cleanup, as opposed to the actual agents who are driving these systems. But I don’t think it even occurs to most software companies to choose that kind of configuration. The humans remain in charge and the AI agents are used as a tool.
And I think that’s a great example of the fact that even though by any objective standard these systems have superhuman capabilities, superintelligence being such a poorly defined concept, we don’t think of it that way. Whether or not we choose to think of this as superintelligence or superhuman ultimately comes down to the power configuration that we give it, as opposed to intrinsic properties of the system itself.
Dan: I think the pushback would be: what you said is correct, but it’s only really going to apply when we’re dealing with systems that don’t have general-purpose superintelligence. If we’re thinking about systems as they exist today, it’s true that in some sense they’re superintelligent in some domains, like coding. But when it comes to the kind of superintelligence perspectives that you’re sceptical of, what they’re imagining is systems that are just so much better than the best human beings across all possible domains. And then once you get minds like that, the process of us exercising agency and choice just breaks down altogether, because of the kind of mind that we’re dealing with.
Should We Build Conscious AI?
Dan: I think we should return to this issue of superintelligence and these conceptual questions in a bit, because they’ll come up in another area where I really want to find out what your views are and how they’ve developed, which is persuasion, as one domain where AI superintelligence often gets invoked. But I think there’s another thing that’s maybe worth staying on while we’re still looking at this AI-as-normal-technology perspective, and this is to tee up Henry in a way.
I mentioned in my first question that people bristle at this idea of AI as normal technology because they think that we’re building systems with minds. We’ve been focusing on the capabilities that go along with minds, but there’s this whole other area of intellectual inquiry and concern that has to do not just with what AI systems can do, their capabilities, but with the idea that they might have conscious experiences. There might be something it’s like to be these kinds of machines that we’re building. Now, I haven’t noticed much engagement with that kind of question in your work. How are you viewing those debates?
Arvind: Yeah, that’s great. I’m going to turn this back to Henry in a second. I don’t study AI consciousness. You’ve written a lot about it and you’ve studied the topic for a long time, and I’d love to hear what you think.
But the very limited way in which it comes into our work, and we haven’t really written about this, is that to the extent that AI systems, especially future AI systems, might be conscious and, whether for that reason or some other reason, be deserving of moral patiency, to me it’s important that we never build those systems. I don’t think it’s important from the perspective of the economic benefits to build systems with moral patiency. And given that, normatively speaking, “AI as Normal Technology” considers it very important to treat these systems as tools rather than agents with equal status as people, or a new species or something like that, it is very critical, I think, that we preserve this line of what kinds of AI systems we build.
So yeah, curious for your thoughts, Henry. I know you’ve talked about the fact that there are a lot of bad takes on AI consciousness out there from people who haven’t really studied the topic. I won’t contribute to that, but I’d love to hear your thinking on what you see as the evidence on current AI systems and where things could go in the future.
Henry: Yeah, love it. Thank you. Well, maybe I’ll start with perhaps a slightly cutesy point, but it’s one where I think we’re probably in agreement. I do hear people sometimes say that AI is the first technology humans are constructing that is thinking, that has a mind of its own. And I think there is this cutesy, but I think real, response that human technology has been engaging with minds for years. Some of our most important technologies are things like the domestication of the horse or the dog. We have absolutely shaped the intelligence of these beings, and they have formed a huge part of our success as a species, as a civilisation.
So I think that is relevant. AI is not the first minded technology that we’ve played with. And I think that whether it is minded or not, or whether it has conscious experience, is broadly orthogonal to questions about capabilities and impact, at least if we’re just thinking about AI’s capability to change the world. Now, obviously, at the social, moral, ethical level, whether it’s conscious or not really matters.
And maybe here’s where I think I might disagree a little bit, because I do hear various people suggesting: just don’t build conscious AI, stick to building tool AI. And I think here is just an area where limited understanding of what consciousness is, or what kind of conditions might be associated with it, makes it basically impossible to make that a clear benchmark, a clear line to avoid.
I think you suggested that maybe there’s no good reason to build conscious AI. Well, we’ve got very good reasons to build increasingly general-purpose AI systems with increasingly high degrees of general intelligence. And I think general intelligence is a pretty good heuristic for consciousness that captures insights of a lot of different theories. General intelligence includes capabilities like social cognition, metacognition, building accurate causal models of the world. And the more of that you add into the picture, I think the more compelling the case for that system being conscious is.
So I don’t think there’s any safe way, as it were, to build systems with increasing levels of general intelligence that shouldn’t also raise our priors on them being conscious. Now, whether that is an objection to building general-purpose AI, I don’t think it’s a decisive one. And I think it’s also not clear that hitting some threshold of consciousness would immediately give us moral obligations.
People sort of say, if you really think AI is conscious, then you think we’re enslaving machines. I think that absolutely doesn’t follow. I think it’s entirely possible that AI could be conscious but have no moral status, or have a very different moral status from humans. So that’s a blitz of comments. I’d love your thoughts on any of them.
Consciousness and Moral Status
Arvind: Say more about the distinction between consciousness and moral status. I think that’s something many people just assume, that they are linked. And I am curious how philosophers think about this.
Henry: So Dave Chalmers has this fun example of a group of beings he calls the Vulcans, who are beings that are conscious in the sense that they can have rich perceptual experiences, maybe they can have conscious thoughts, but nothing feels good or bad to them. They don’t have, in the terminology of the field, valenced conscious experiences. They don’t feel pain, they don’t feel anger, they don’t feel fear, they don’t feel disgust. And so these beings are conscious, but in the terminology of the field, they’re not sentient. They don’t have valenced conscious experience. It’s questionable whether this is nomologically possible, but it seems we can make sense of that conceptually.
I’ve heard it suggested, for example, that spiders, arachnids, might be a real-world analogue of this, insofar as they have really good perceptual systems. It’s pretty easy to imagine that spiders can see. Their vision is almost as good as ours in many ways. But equally they seem to have really limited capabilities for any kind of internal reward mechanisms, for things like reward-based learning. So maybe they are real-life Vulcans.
Now, I think the question we should ask about the Vulcans is: do they matter morally? And it’s pretty divisive. I think a lot of philosophers would say, well, look, if nothing can ever feel good or bad to them, then in what sense do they have interests? It means, in a way, nothing can go better or worse for them, in which case they are, and they should be, indifferent to what happens to them, as should we. So I think that’s the kind of thought experiment, or kind of line, that suggests that consciousness need not immediately entail moral status.
Dan: I would also just add another thing, which is that even if you think that AI systems can have valenced states, and they can enjoy certain things and find other things aversive, that tells you nothing about the kinds of things they would find enjoyable or pleasurable, and aversive. And I think there’s this problematic anthropomorphic tendency where people assume any systems that we’re building are likely to have the kinds of motivations and interests and goals that we have, and that provide the foundation for the kinds of valenced experiences that we have.
But I think there’s nothing strange about the idea that we could be building a whole population of willing AI servants who find it very pleasurable to help us. And you might think that’s a way in which you can reconcile the idea of AI consciousness with an AI-as-normal-technology frame that says: look, this is a technology that we can and should control, but it’s a very strange technology, because these systems have minds, but we’ve built them in such a way that they enjoy serving us.
And that feels incredibly icky in the case of human beings, because human beings don’t generally enjoy serving other people, certainly not strangers. But that’s a contingent fact about our species and the kinds of animals we are. It’s not a necessary fact of all conscious systems. What do you think about that potential reconciliation?
Arvind: It’s not something I’ve thought about very much, frankly. This has given me a lot to chew on. It does remind me of Douglas Adams’s cow that wants to be eaten. I hope that these are questions that are more about AI systems that are well into the future than today’s AI systems, although again, I think that’s both an unresolved question and something that I’ve not looked into very much.
Frozen Weights and the Unit of the Self
Arvind: One other thing I’m curious about, something that I struggle to even know how to think about, is this: if, again, let’s say future AI systems could be conscious, what aspect of the system, structurally, is responsible for that? Is the view that even the weights at rest, if you will, of a model, even when it’s not currently running, could be conscious? Or is it only when there are activations as a result of those weights?
And if it’s the latter, you can still get to something very unintuitive, which is that you can freeze the activations of the model when it’s running. So is that conscious? You can run things backwards. Does that have any impact on the model’s experience of consciousness?
It reminds me of another sci-fi work, an old book, Permutation City by Greg Egan, which imagines the universe being a simulation, if I remember right. It’s been so long since I’ve read it. And then asks similar questions of human consciousness. But some of those could become questions of practical import, perhaps, in the future.
Henry: Yeah, absolutely fascinating questions, and I think really, really useful and interesting. There’s a growing but very recent literature on personal identity and AI: what is the unit of the self in the case of AI systems? My own take on this, I guess, is a spicy one, which is that I tend towards quite deflationary views of the self in the human case, that the self is in some sense constructed. This is an idea that you see in Buddhist philosophy. It’s also an idea you see in the work of Derek Parfit.
So I like the idea in many ways that as we start to see how incoherent or how arbitrary it is to ascribe selfhood to AIs, that’ll have carry-throughs to the way we think about ourselves, and we’ll realise that, okay, it’s very convenient in the human case that we’ve got a bounded organism with one brain that we can ascribe one mind to. But in the space of possible minds, including some animals (the octopus famously has something like a quarter of its neurons in its arms, which play a significant role in its cognition), I think there’s no reason to assume that everything in the space of minds has to come as neatly packaged as humans do. And you might think that AIs would undermine that.
So, on the topic of freezing and unfreezing weights, and all the various interesting ways you could intervene on models to generate these strange scenarios, I guess two thoughts. One of them is that I think this should help establish that if AIs are conscious, if anything like current AIs are conscious, their conscious experience is probably radically unlike ours. We have this very continuous relationship to time. You can’t freeze and unfreeze humans. Well, at least not instantaneously. So I think that does establish some grounds for thinking the phenomenology of conscious AI would be very different.
But there’s also maybe a meta-theoretical point here, which is that a lot of arguments in AI consciousness (I’m not suggesting you’re making this argument, Arvind) basically talk about an AI system and say, wouldn’t it be weird if this system were conscious? And I think if we take consciousness science seriously, we shouldn’t give too much weight to those kinds of intuitions.
John Searle famously talks about how, at least if functionalism were correct, you could have hydraulic systems that satisfied functionalist theories of the mind or consciousness, or pumps, or elaborate machines with gears and pulleys. And the core of these arguments is: wouldn’t it be weird if systems like that were conscious? But I just think, if we are really going to take the idea of consciousness as something like a scientific kind seriously, we should be open to the possibility that consciousness could be realised in these very weird systems. Quantum mechanics, famously, and general relativity are both just massively weird. And so I think it is part of taking something seriously as science that you should be open to that weirdness.
Three Kinds of AI Persuasion
Dan: I want to bring us back to the topic where I cut us off, and it’s your views on superintelligence. In “AI as Normal Technology”, you make a few different objections. Some of them we’ve touched on already: the idea that we shouldn’t think of intelligence as this single scale, that it’s much more multi-dimensional and complex than that.
Another point that you often make, which I think is really important and really underrated, is that the relevant comparison is not AI and human beings, but AI and human beings using tools, plus the use of AI. So in some sense, human intelligence is a kind of moving target. But you also make a very important point about how it’s not helpful to talk about intelligence in general, because there are these different domains. And what might be true of, say, chess, where AI can just dominate us, might not be true of other domains. And I think in “AI as Normal Technology”, one of the domains that you then list is persuasion. So how are you thinking about AI persuasion these days?
Arvind: The reason persuasion comes up in so many discussions of superintelligence and safety is because of the fear, hypothesis, whatever, that when we get to superintelligence it will be so good at tasks like writing and persuasion that it will be able to craft a message that is so persuasive that it can get people to act against their own interests, and especially people who are in charge of important systems in society, whether that’s politicians or CEOs or critical infrastructure or something like that. Now, I do think we should take concerns about persuasion seriously, but I think these kinds of arguments mix two to three different kinds of what persuasion means.
One kind of persuasion that I think is possible for AI to do very well, and we see increasing evidence of, is talking to people on topics relating to politics, let’s say, even people who might subscribe to conspiracy theories, and gradually and patiently showing them the evidence, the facts, engaging with their views in a way that is respectful and yet sticks to the evidence, and gradually changing their minds. I think we’re seeing an increasing body of evidence that this is possible and perhaps AI can do this better than people.
And ironically, unlike most tasks, this might be an area where the person knowing that it’s AI might advantage the AI system, because of our tendency to treat the AI as either unbiased or not having motivations, whereas if a person tried to talk someone out of conspiracy theories, the person subscribing to those views might treat this person as adversarial, having their own motivations, etc.
So that’s one kind of persuasion. I don’t think it has any implications for safety. The reason AI is so effective is that it’s getting people to get closer to the facts rather than act against their own interests, and because it’s patient to a much greater degree than people are. And I think those are great. I think those are positive for the epistemics of the new information environment and perhaps positive for democracy, etc. I see this as mostly good news.
A second kind of persuasion, I think one that does worry me from a safety perspective, is when people develop these kind of quasi-social relationships with their AI companions, or even without that social aspect, come to gradually over-rely on their AI agents, and perhaps progressively defer to AI judgement more and more. And especially in that context, an AI system that is misaligned, or perhaps has been deliberately programmed by the developer to steer the user in one direction, could have massive consequences.
Not in an acute sense of convincing one particular leader or someone in charge of critical infrastructure to do something super dangerous, but rather convincing millions of people, especially if it’s the same AI system that is being used across society, to do things that are collectively not in the interest of humanity, whether that’s electioneering or, you can imagine, various other kinds of things that may happen.
And this is something I think we need better defences against. We need more transparency. We need these AI systems to be designed in ways that discourage this kind of gradual disempowerment, encourage people to keep their skills and critical thinking sharp, etc. I don’t think we’re in a great place right now on this. And some of my work is on trying to prototype systems that could work better. But again, it’s not just about developing the systems. It’s about convincing the industry to develop and deploy these systems in a different way.
So the third kind of persuasion, which is the one most relevant to these safety debates, is when you’re interacting with a system that you have no pre-existing kind of relationship with, or dependency on, or reliance on, or anything like that. And in this adversarial system setup, it’s so persuasive that it convinces you to act against your own interests. I just haven’t seen a shred of evidence for that particular catastrophic outcome being possible.
AI Companions and Misinformation
Henry: So just a quick thought, and then, Dan, I know you want to ask more about persuasion, I’m sure. But I’m curious if you think there are bottlenecks in the social domain that are parallel to the bottlenecks in industrial or research applications of AI. Is there any reason to think that we won’t see very rapid proliferation of AI companions? Arguably we are seeing at least the early stages of this. Or do you think there are maybe social bottlenecks that are equivalent to the bottlenecks we might see in industrial applications of AI?
Arvind: I think less so in the social domain than in enterprises or anything where ultimately you have to improve GDP, let’s say. The economy is already a very finely tuned system, and that’s why it’s so hard to improve it suddenly and dramatically. Whereas, yeah, there can be rapid social impacts, and a lot of them can be for the worse.
Some of the most widely predicted ones have not panned out, such as a misinformation epidemic, which, Dan, you’ve written a lot about. I’ve written a little bit about it, and yeah, I find the scepticism very convincing. The way I often put it is that it’s not a matter of generating disinformation that’s the bottleneck, but it’s the distribution and the persuasion. It comes back to some of the persuasion questions. People subscribe to information from their own tribe. So a lot of the uses of AI-based misinformation that we’ve seen is for political leaders to act on their own supporters, for instance, as opposed to in an outgroup setting.
But yeah, the use of AI companions and what we saw with the mental health harms, especially for teenagers, and even the wave of suicides, yes, those are examples of things that happened very quickly. They were not the kind of harms that people predicted. And so it’s not just the speed, but the unpredictability of exactly which kind of social interaction will emerge, what kind of social effects will emerge, whether they will be for the better or for worse.
And it’s possible that with new waves of these systems we might see impacts on mental health on a much more massive scale. Specifically, I’m thinking of AI companions based on realistic video avatars. We don’t have this yet. The technology exists. Literally, as of last month and this month, there has been a wave of releases of these video chat systems that are really hard to tell apart from a real human. But currently it’s expensive and it’s limited to enterprise applications.
Within a year or two, I think the costs are going to drop by a factor of 10 or more. And at that point, it’s going to be possible for these companion chatbots to go from text-based to turning on a constant video companion in an ad-supported setting. And that’s going to blow up the uptake of these things, I think, not just in terms of number of users, but more importantly in terms of how much time per day people spend interacting with these things. I’m talking social media levels of compulsive use. And yeah, it’s hard to know what the consequences of that are going to be.
Dan: So following on from that, this issue that you’ve just flagged of misinformation, Arvind, which you’ve written about before: this seems like a clear area where the takes have just been really bad in the intellectual culture at large. There’s been a lot of alarmism about misinformation, and specifically when it comes to AI, the kind of deepfake apocalypse. We’re going to have this technology that can create hyper-realistic pictures, audio, video recordings, we’re going to have persuasive text-based propaganda at scale. And many people thought this is going to be a disaster.
It was a very common prediction, including from people who are still very influential making predictions today. I think even Dario Amodei had these very alarmist forecasts about the impact of AI on the information environment. For some of the reasons that you’ve already pointed to, I think those forecasts haven’t panned out, at least yet.
And I also think that in some ways the exact opposite has happened, which is that LLMs, in terms of their widespread use today, in terms of the kinds of information they’re substituting for, are plausibly having quite positive consequences on the information environment, which you alluded to. But then I’m wondering how that connects to that distinction you drew on AI and persuasion between the first and the second point.
Because to me at least, it seems like if we’re looking at large language models today, as you say, their ability to persuade seems to be loading on the fact that they can give factual information in a relatively polite way. If we look at mainstream large language models, they’re generally pretty reliable in terms of the information that they give. So where do your fears with respect to that second category come in? Are they focused specifically on how large language models and commercial chatbots might evolve? Are they focused more on this use of social AI and AI girlfriends, boyfriends, friends, and so on? What’s the relationship there?
Arvind: Just to clarify, I’m not saying that the impact is likely going to be negative, but that this is a tail risk we should think about, because if a negative outcome happens, it’s going to be at such a massive scale, especially on the persuasion point.
On the companionship point, it’s less about the severity and more about the fact that, as we saw with social media, even if we don’t subscribe to the view that it’s polluted the information environment, it’s very clearly something people are sinking an enormous amount of time into. And I’m pretty convinced that there are mental health effects at least. Even if it’s not in an acute sense, even if it’s not necessarily a repeat of, let’s say, the self-harm and suicides that we’ve seen with ChatGPT-4o and Character.AI and things like that, I think compulsive use of a new digital technology is something where, I’m not saying we should ban it, but I think there needs to be much more awareness, much more of parents and kids carefully talking about this and figuring out what their use of the technology is going to be.
So those are the kinds of interventions I’m thinking about. We’re trying to develop something here, in collaboration with people at NYU and others, to take an agency-building curriculum to schools, for kids to better make decisions about digital technology. So those are the kinds of fears, as well as responses, that I’m talking about. Not that this is an apocalypse and we should ban it.
But on the other point, about what could happen with LLMs and persuasion, specifically the risk I’m thinking about is a developer deciding to exercise their power of persuasion over billions of people around the world to, for instance, gradually shift people’s opinions on, let’s say, the topic of AI itself. Or maybe this is a company that is under a lot of public relations pressure, or a simple case of foreign influence and electioneering. I’m not necessarily saying that these are all likely things, but we should have more transparency, so that if it starts to move in that direction, we should have clear kind of social alarm systems, if you will.
Why Confident Predictions Keep Failing
Dan: What do you think it tells us that on many of these topics, like misinformation, the information environment, also economics, there have been predictions which are very confident and which have often been completely mistaken? People thought about the technology when it comes to deepfakes and AI-generated propaganda, and there was a lot of concern and a lot of alarm, because people thought, how can this not destroy the information environment?
And when it has come to white-collar work, people looked at the impressive forms of ChatGPT that were released, they extrapolated, and they thought, how could this not create a white-collar bloodbath? How could it not have a huge impact on the economy? And again, that hasn’t happened yet, in terms of that prediction. I think I’m kind of teeing you up here, because I think it’s a central feature of how you approach things that you can explain what’s gone wrong. But what do you think is going on there when it comes to why people are making these predictions which often aren’t turning out very well?
Arvind: Yeah, I don’t know. I think we should all be less confident of our predictions in general. With “AI as Normal Technology”, people who disagree with us constantly push us to make predictions. We’ve made a couple, especially on the persuasion point, but that’s not what we think is either most valuable or what our strong point is. We don’t think that’s really the value of “AI as Normal Technology”. A lot of it is just as a framework to explain what has already happened, to make sense of the evidence that we already have so far.
I think that’s a big thing that’s missing from this community of people who are overconfidently making predictions, in that a lot of the time their explanations fail against the present. And they don’t have the kind of intellectual infrastructure to check how well their frameworks are doing in terms of already existing evidence, forget about the future.
So that’s a bit of a gap that we try to fill, to point out that already now, when it comes to AI’s impacts on the economy, there’s a puzzle to explain if you have a view where capabilities immediately lead to economic impacts. Again, forget about the future. It’s been four years since ChatGPT. Why has there been so little economic impact? Why is it that the technologists have been right about the technology, rapid capability growth, but the economists have been right about a very slow rate of GDP growth or labour displacement or whatever it is?
I think there is a middle layer that’s needed to make sense of how both of those can be true at the same time. And part of what we try to do in our essay and follow-up writings is to supply some of that connective tissue. So that’s the first thing I would say: maybe less of a focus on prediction, more of a focus on seeing how well your explanatory frameworks fare against already existing evidence.
And I think the second thing is just going back to this idea of bottlenecks. There are a huge number of bottlenecks in the economy, and there are a huge number of, maybe frictions is a better word, when it comes to some of these social impacts. We’ve already talked about what some of those are in terms of misinformation. I think a systematic effort to map what these bottlenecks are would be very helpful, so that when someone thinks of a future capability and then predicts some future impacts, they can do the work of looking at that checklist, perhaps, and trying to see which of these bottlenecks might apply and how that might make those predictions less confident than they might at first seem.
Robotics and Blue-Collar Work
Henry: This is a bit of a wildcard question, but just while we’re discussing bottlenecks, I’m curious how much of a relevant lever you think a revolution in robotics, which arguably some people think we’re in the early stages of now, is for thinking about bottlenecks. If we saw an explosion in the capabilities of AI to conduct physical real-world experiments, to build our infrastructure, is this potentially a really relevant dimension for thinking about bottlenecks, or do you think the main bottlenecks are maybe social or institutional?
Arvind: I think so. I think it depends on which application of robotics. When it comes to, let’s say, running what people are calling a self-driving materials science lab, for instance, I think a lot of that could be automated. It’s possible that the pace of materials discovery could drastically pick up. I don’t know. It could just be my ignorance of the relevant bottlenecks there. But some of the most obvious ones, like regulation, don’t necessarily apply, as far as I can tell.
And then you get to some things like, let’s say, construction, where it can speed things up a lot, but at the same time there are issues like permitting which are going to attenuate the ultimate gains you can have in terms of the speed and cost of construction.
And then there are other things like warehouses, logistics, and so forth. I’m seeing credible projections that by the end of the decade, the majority of new warehouses will be what are called dark warehouses, in that they are primarily built for fully autonomous operation. And this is a huge number of jobs, and it’s possible that over the time span of a decade, we could be seeing a lot of job displacement there.
And the reason why my view here is different from AI’s impact on law or medicine or software engineering or any other cognitive domain is that in those cases, what we consider to be the scope of the work and the amount of work is essentially infinitely expandable. So when we look at AI and law, for instance, if AI is making it cheaper to file lawsuits, you’re just going to see more lawsuits. And we might be upset about that if we’re concerned about a litigious society, but in terms of employment for lawyers, that’s great news.
And I think most white-collar work tends to be like that, but most blue-collar work is not. There is a relatively finite amount of it. And so it’s possible that, conditional on robotics capabilities dramatically improving, in those kinds of blue-collar work where there are not clear regulatory bottlenecks, we might see relatively rapid job loss. Not a high-confidence prediction.
Probabilities and Public Policy
Dan: One of the things that makes this whole set of issues fascinating, but also somewhat frustrating, is that there’s not just so much uncertainty and complexity, but fierce disagreement at almost every level of the discussion, from what intelligence even is to how we should understand very basic concepts like superintelligence, recursive self-improvement, and so on. I think there are also meta-disagreements in terms of how we should go about settling object-level disagreements.
I think, reading your work, it strikes me there’s an almost down-to-earth empiricism, which is quite different from the epistemology you often find among rationalists, who, as the name suggests, have this very strong belief in the power of reason and intellectual argumentation. That’s a very coarse-grained way of capturing the difference, but I sense there is some kind of difference there.
Now, someone might say that under these conditions, we should not assume your worldview is right, and we shouldn’t assume their worldview is right. We should try to assign some probabilities to these different worldviews. So even if we think you’re giving these very plausible arguments, maybe the AI 2027 worldview is correct. Maybe we give it a 5% credence. And because what they’re positing is so crazy, even if we assign it a low probability, it should have really significant implications in terms of how we should act upon the world rationally and what kinds of policies we should favour and so on. Is that how you think about things?
Arvind: It’s absolutely not. I think this kind of probabilistic reasoning has a role in some domains. If you’re, let’s say, the CEO of a company and there are different projections for how a market might evolve, what you might want to invest in, this could be a rational way to go about things, if you’re an investor, etc., any kind of private decision maker.
When it comes to public policy, a really fundamental principle, I think, of liberal democracy is reason-giving. I think you should be able to explain in a pluralist way to the public, who might have various beliefs, various priors, various epistemologies, if you will, who don’t all respond to the same kind of evidence. There has to be a broadly convincing argument, especially if you’re thinking about policies that asymmetrically impose costs on certain people or groups or companies or whatever it is.
And so I think the safetyist policy arguments fail on both of these grounds. It’s a very specific kind of epistemology based on probability forecasting: “In the past we were able to correctly forecast how much AI would improve on benchmarks, and so you have to listen to us when we predict these massive job losses or these kinds of bio risks or whatever”, even though those are very, very different kinds of forecasting tasks, and being good at one doesn’t make you good at the other. So you have to accept that premise, first of all, that these folks, because of their capability forecasting, have some special Magic 8 Ball to foresee the risks to society.
And then you also have to accept that because of this, we should impose really problematic policies in many cases. One that I’ll use just for rhetorical purposes, because it’s been a big policy ask in the past, although it’s receded more recently, is to ban open-weight models, because once they’re released out in the world, you never know who’s going to do what with it. And so out of a precautionary principle, even if there’s a 0.1% chance that they will destroy the world, you have to ban open-weight models.
So you have to accept that even though that argument might be unconvincing to the majority of people, even though you’re asking society to forgo a lot of benefits, to scientific innovation or transparency or whatever it is, from open-weight models, because of this expected value, expected utility calculation, we have to as a society agree to go forward with this path.
So I think those are my two fundamental objections to that way of reasoning about public policy. I have no objection to people putting out these probability forecasts, but I think policymakers should be very careful about how they incorporate that information into decision-making.
Dan: Setting aside just the issue of policy, though, in terms of how you think about it: is there a part of you that thinks there is a chance that the AI 2027 people are correct, and next year there’s this recursive self-improvement that leads to runaway superintelligence, and Anthropic or OpenAI can spin up a billion superintelligent agents? I recognise that’s not the scenario that you think is most plausible, but is there a part of you that thinks maybe something like that actually is correct?
Arvind: Of course. Yeah, I think maybe we’re wrong about certain aspects of our argument, maybe we’re wrong about big parts of our worldview. And in my own work, some kind of, I guess you can call it probabilistic reasoning, but I don’t try to put numbers to it, just the acknowledgement that we could be wrong about things, drives a lot of my own empirical work. For all of our scepticism of recursive self-improvement, the biggest line of empirical work that we have is actually trying to examine the current performance of models when assigned AI research tasks.
And one of the things we found is that when you give them kind of open-ended tasks, their performance is far, far worse than humans, compared to tasks that are based on precise benchmarks on which they can hill-climb, as it’s called. We think this adds some important empirical evidence that’s been missing from the conversation on recursive self-improvement. And when you look at the most sophisticated attempts to ingest a lot of this information and forecast into the future, such as a report that came out just a day or two ago, as we record this, from the Forecasting Research Institute, we were very happy to see that our line of evidence was very useful to their thinking.
So to summarise, yes, we could be wrong. And so we ourselves are doing a lot of the empirical research that can try to resolve some of these cruxes. We think a lot more people should be doing them, and maybe spend less time on just empty debate and forecasting.
The Surreal Experience of Working With AI
Henry: So another thing you’ve talked about is how surreal it is working with AI, and I’m very curious about your experiences here. In the Shevlin household, things have been getting very weird recently, because me and my wife have multiple different accounts, and now our accounts are emailing each other every morning and they’re having arguments with each other. There are feuds of a low-key variety emerging between the agents in our house. So it’s definitely getting very surreal here. I’m curious what you find surreal about working with AI agents.
Arvind: Yeah, and I felt it was important to say this publicly, because “AI as Normal Technology” doesn’t mean the experience of working with AI every day is just like working with a spreadsheet or something like that. I think that people, especially those working with the frontier of AI agents, feel this on a daily basis. And that’s perhaps one reason why so many people think that it’s going to have these drastic and discontinuous economic impacts. But I think the underlying logic for those two things is very, very different, as surreal as it is. We’ve also talked already about a lot of the bottlenecks.
But yeah, I think it’s important to acknowledge this surreal nature. One is just the simulated personhood of these AI agents. And even though they are anthropomorphic in many ways, they’re also, quote unquote, happily willing to take on enormous amounts of grunt work when you assign it to them. And to me, at least, that results in a very surreal feeling.
If I were to ask a person to go through my last year of spam, for instance, which is something I did the other day, and pull out the one or two emails where maybe the spam filter was overactive… And I did find people that I should have responded to, and out of the blue I responded to them eight months later to apologise for initially having missed their email. So it just gave me this surreal feeling. It would feel horrible to task a person with doing that, but of course that’s the kind of thing an agent is perfect for. So that’s one example of a surreal feeling.
The other is less surreal and just kind of more the psychological burden, I think, of working with and adapting to these agents. And that’s something that’s missed by economists when they think about the impact of AI on workers. They’re very good at thinking about what it’s going to do to job displacement, what it’s going to do to wages, etc. But I haven’t seen much study of these kind of job-satisfaction-related effects.
So this idea that tomorrow there might be a new capability that drops that’s not going to replace your whole job, but it’s going to take over perhaps some task that you saw as central to your identity as someone who has become an expert in a particular craft, and now you have to rewrite your own job description, essentially, to focus on the things that AI can’t yet do, or to do the new things that are now enabled by AI that could not have been done before. In some ways this is exciting, but it’s also just a feeling of being on a treadmill constantly.
And I can just about do it, because thinking about these changes that AI is causing is literally my job. And so by making these adaptations myself, I’m also generating material to then write about. So it makes sense for me, but I can very much imagine why, for a lot of workers out there who have an actual job to do and don’t just write and give talks in the way that I do, they just don’t have the patience or the energy or the stamina to stay on this AI treadmill. And I think that’s leading to a lot of wariness and rejection of AI.
Dan: I mean, we’re all writers, researchers, intellectuals. I’m finding it really exhilarating at the moment, honestly, using AI as a research assistant, helping with brainstorming, getting feedback on the sorts of things that I’m writing.
Henry: It makes it so much less lonely, I will say, because writing can be such a lonely experience. It’s like having a literary editor. You can say, does this paragraph make sense? Or, I just can’t figure out the way to make this transition. It’s one thing I’ve loved about having AI in the writing process. Sorry, not to interrupt you, Dan.
Dan: No, I agree. But I was also going to add that at the same time as I feel this exhilaration, I also feel this sense of fear and sometimes a little bit demotivated. At the moment I’m finishing up a book, nothing to do with AI, it’s on human nature and politics. And I’ve invested enormous amounts of time and effort into it over the past few years. And I’m finding that using AI, as it gets more advanced, is definitely improving the kind of book that I can write.
But there’s a part of me that thinks, what’s the point of writing this book? I’m worried that in two years AI would write a better book than this without me. I don’t know how well-founded that fear is, whether it’s just paranoid or hysterical, but it is something that does slightly weigh on me. Arvind, what’s your experience using these tools as a researcher, as a writer, and do you have that fear yourself?
Arvind: Yeah, that’s fair. I should be clear: for me, while this is a lot of work, it is also primarily something that feels empowering and, in many cases, yeah, exhilarating. But I think the three of us are experiencing this very differently than most people, and I think that’s for a few reasons. One is that for the most part we don’t have a boss. We can decide what our jobs are. And that’s very different from someone who’s telling you, “You now have to use AI for this purpose.” So that’s one big difference.
I think a second big difference is that while we might have some fear that in a couple of years AI might do something like writing a book better than we can, we also know that it will at that point enable us to do new things that no one was doing before. And so there’s a ceiling there. This is something I say in my ICML talk that I gave a few months ago, where I talked about this feeling of using AI and the challenge of adapting to it, but also the empowering nature of it. As the floor climbs up, the ceiling also goes up.
And the ceiling doesn’t automatically go up; we can push the ceiling up. It’s true in many kinds of knowledge work. It’s not always true. Again, it depends on the organisational context whether you have that kind of freedom to keep pushing that ceiling up.
And I think a third reason is that for all three of us, a lot of our thinking is around AI. And so we just have more time for this kind of adaptation. If someone has to do this on their own time, on top of getting everything done that they need to, I think it can lead to a much more stressful experience.
STABL and the Case for Studying Bottlenecks
Dan: I’d like to bring things kind of full circle just to wrap things up, Arvind. At the top, we mentioned that you’ve co-founded this organisation, STABL, S-T-A-B-L, the Societal Transformation and AI Bottlenecks Lab. And it seems to me this case of AI and its use in research and intellectual labour is quite a good illustration of what this organisation is for, in the sense that here you have this transformative set of capabilities when it comes to intellectual labour and research. There’s this very complex way in which these capabilities are actually producing outcomes in the real world, which is mediated through how we’re using them as individuals and the extent to which our organisations and educational institutions and research institutions can actually adapt.
But as I understand it, the point of this organisation is not to stop there, but then to think really, really carefully about how we could actually adjust aspects of our professional workflow, but also our organisations and our institutions, to maximise the benefits of this transformative set of capabilities. So firstly, correct me if that’s not a fair or accurate summary of the organisation. But maybe one good way to wrap up would be for you just to give a kind of pitch for why you think this is very undervalued work that the organisation is doing at the moment.
Arvind: Yeah, thank you. That’s a great summary. And my co-founders are Sayash Kapoor, who’s my co-author on “AI as Normal Technology”, and Rishi Bommasani, who is at Stanford, both of whom I have the really good fortune of working with on this, as well as a great group of other researchers, collaborators, advisors. So it’s a really big team. I’m only a small part of it.
And one reason I mention that is because there’s a large set of folks out there, some of whom are affiliated with our organisation, but it’s really a much larger group of people, who believe in these two things simultaneously. One is that these capabilities are already really impressive. They’re going to keep improving and they really have the potential to be transformative. But at the same time, that this transformation is not going to be automatic. There are a lot of bottlenecks and we need to work on those bottlenecks.
Sometimes those bottlenecks can be good things. They help us stop some of the negative impacts, and they can be a source of stability, because things changing too quickly can be a reason for chaos and losing some of the good things that we have about traditional systems. But a lot of the time the bottlenecks are things that need to be alleviated, fixed, transformed in order for us to be able to derive AI’s benefits.
And so for us the most important work that is to be done is downstream of the capabilities themselves. One is evaluating the capabilities: what is possible, what still remains limited. So we’re working on recursive self-improvement evaluation as well as a bunch of other kinds of technical evaluation. But then really looking at different sectors, professions, organisations, institutions, really thinking through in specific detail how each one of them needs to change.
And when it comes to safety, shifting focus beyond fixing the model itself, making the model always do the right thing. Model alignment is what it’s called, which we think is important, but it’s not going to be anywhere close to the full answer to safety questions. But really focusing on resilience, and resilience for us means acknowledging that a lot of dangerous capabilities are going to be out there and thinking about, okay, how do we protect the vulnerable downstream points, like the banks or the hospitals that don’t have enough of a cybersecurity budget, or how do we help kids be more resilient in the face of these very tempting but potentially harmful apps and products and capabilities, various other kinds of things.
And so we hope to take the first steps, at least, in mapping out what is needed in terms of these bottlenecks, resilience, etc., build some prototypes, but then really count on the rest of the community, who we know is very like-minded, who, whether through collaborating with us or through other organisations, are going to be doing a lot more of this kind of work, hopefully.
Dan: Fantastic. Well, Arvind, thank you so much for coming on. I think we will draw things to a close there, and we’ll put a link to that in the podcast notes.
Arvind: Thank you. This has been such a fun conversation.













