Henry and I discuss controversies surrounding Artificial General Intelligence (AGI), exploring its definitions, measurement, implications, and various sources of scepticism. We also touch on philosophical debates regarding human intelligence versus AGI, the economic and political ramifications of AI integration, and predictions for the future of AI technology.
Chapters
00:00 Understanding AGI: A Controversial Concept
02:21 The Utility and Limitations of AGI
07:10 Defining AGI: Categories and Perspectives
12:01 Transformative AI vs. AGI: A Distinction
16:15 Generality in AI: Beyond Human Intelligence
22:13 Skepticism and Progress in AI Development
28:42 The Evolution of LLMs and Their Capabilities
30:49 Moravec’s Paradox and Its Implications
33:05 The Limits of AI in Creativity and Judgment
37:40 Skepticism Towards AGI and Human Intelligence
42:54 The Jagged Nature of AI Intelligence
47:32 Measuring AI Progress and Its Real-World Impact
56:39 Evaluating AI Progress and Benchmarks
01:02:22 The Rise of Claude Code and Its Implications
01:04:33 Transitioning to a Post-AGI World
01:15:15 Predictions for 2026: Capabilities, Economics, and Politics
Transcript
Please note that this transcript is AI-created and may contain minor mistakes.
How Close Is AGI?
Dan Williams: Welcome back. It’s 2026, a new year, a big year for AI progress, an even bigger year, dare I say it, for this podcast. I’m Dan Williams. I’m back with Henry Shevlin. And today we’re going to be talking about one of the central, most consequential, most controversial concepts in all of AI discourse, which is AGI, artificial general intelligence.
So AGI is written into the mission statements of the leading AI companies. OpenAI, for example, states that their mission is to ensure that artificial general intelligence benefits all of humanity. We also constantly see references to AGI in the media, in science, in philosophy, and in discourse about the dangers, potentially catastrophic dangers, of advanced AI. And yet, there is famously very little consensus on how to even understand this concept, let alone measure our progress towards it.
Is it, for example, a system that achieves something called human level AI? Is it a system that can do any task or at least any intellectual task that a human being can do? Is it a system that performs extremely well on tests, on benchmarks? Or is it, as some people suggest, a deeply confused pseudoscientific concept? So for example, the influential cognitive scientist Alison Gopnik has said, there is no such thing as general intelligence, artificial or natural. Jan LeCun, one of the most famous AI researchers in the world, says this concept makes absolutely no sense.
But if that’s the case, what should we make of people making predictions about when we’re going to reach AGI, perhaps in the next few years? How do we make sense of rapid AI progress? What are we making progress towards? Moreover, what do we make of people, smart people, who claim we’ve already reached AGI, that we’re living through the post-AGI world?
So these are the topics that we’re going to be focusing on today. What is AGI? Is the concept coherent and useful? How do we measure progress towards AGI if we take this concept seriously? And what happens when or if we reach AGI? At the end, Henry and I are also going to be giving some predictions about how we expect AI to develop over the course of this year.
Okay, so to kick things off, Henry, AGI, how do you understand the concept? Are you a fan?
Henry Shevlin: I am a cautious fan of AGI as a concept. I think it’s an imperfect concept and can be very vague or defined in various ways. But at the same time, I think it serves as a useful reminder that we are heading towards an era, in my view, of genuinely transformative capabilities in AI systems. And so when we talk about AI revolutionizing science, AI revolutionizing medicine, AI revolutionizing the future of work, I think AGI is often a useful shorthand for talking about the point at which we start to see really massive changes in these domains.
That said, I do have some sympathy for the worry that this is not a particularly coherent concept. So I think we’ve seen commentary in the media recently saying, look, we don’t really understand what intelligence is, and therefore the very idea of AGI is ill-defined.
What I would say there is that I think we don’t need to understand exactly how human intelligence works in order to recognize when we’ve exceeded human capabilities in certain key ways. And in the same way, we don’t necessarily need to have a perfect biomechanical model of how birds fly in order to build planes that can fly faster than them. So I think even with some empirical questions or some conceptual or definitional disagreements about what intelligence is, what human intelligence is, it could still be the case that we’re well on our way to exceeding the capabilities of human intelligence across the board with AGI.
One thing to quickly flag though is AGI is kind of canonically or classically defined as systems that are equal to human level performance across all domains. I think tacitly this is often restricted to sort of economically and scientifically and cognitively relevant domains, right? So I think if we had systems that were sort of at human level or above in pretty much every cognitive task, but they couldn’t smell or had limited ability to do certain kinds of fine-grained motor tasks, perhaps, I think that wouldn’t disqualify us from characterizing those systems as AGI. If they’re doing better science than a human, if they’re winning mathematics prizes, if they’re Nobel’s, if they’re doing 99% of current jobs in the economy, it’s not going to be a deal breaker whether or not they can tell Sauvignon Blancs from a Chardonnay with a sniff.
Dan Williams: Yeah, although on that point, I think there’s a question here, which is, should we expect a system that can out-compete human beings when it comes to what are thought of as purely cognitive tasks, if it doesn’t have the kinds of competencies that go into, for example, folding laundry, making toast, et cetera. So the idea that you can draw a nice distinction between purely intellectual tasks of the sort that you can perform on a computer, and let’s say what I thought of as sort of non-intellectual tasks, of sensory motor tasks. I think that’s a kind of interesting question in and of itself.
Doing a bit of reading around AGI for this episode, it seems like a lot of the definitions about what AGI is splinter into sort of three different categories. And I’ll be interested to hear what you think about this way of taxonomizing area.
So some people seem to understand AGI basically as a kind of placeholder for whatever AI happens to have really transformative consequences. So it’s like, AGI is just a term for transformative AI, whatever form that transformative AI actually takes. Other people seem to understand it with this concept of human level AI or something similar, where they’re sort of using human intelligence as the thing relative to which we should understand the concept of AGI. And that I think for reasons we can probably get into, I can kind of understand what they’re getting out there, but I think there are all sorts of reasons to be skeptical about that concept. And then there’s a third category of attempts at understanding this concept where you’re just understanding it in terms of kind of abstract capabilities, right? And it might in fact be the case that human beings exhibit or instantiate these capabilities. But the idea is you can specify what these capabilities are independent of thinking about the specific form that they take in human beings. So things like the flexibility and generality of problem solving ability or capacities for continual learning and self-directed learning and autonomy and so on. So it’s like transformative AI understood in terms of impacts, you’ve got kind of human level AI where it’s a system which in some ways has capabilities that are like the sort of ones that human beings have, or you’ve got just a kind of pure capabilities understanding.
Does that correspond with how you’re thinking of this area? Would you add any other categories to that?
Henry Shevlin: Yeah, I think that’s really helpful. I guess a fourth category you might add, it’s a bit of a misnomer to call this category AGI. But I think in practice and a lot of discourse, sometimes people use AGI to refer to something like the singularity or some kind of recursive process of intelligence self-improvement. At which point, AGI functions basically as the same as the idea of artificial super intelligence. I think that’s probably not a maximally helpful way of thinking about AGI. I think it is helpful to distinguish between AGI and sort of the singularity or recursive intelligence explosions. But in practice, that’s what some people mean, I think, they talk about AGI.
Dan Williams: Yeah, just to add a footnote to that. So this idea of an intelligence explosion, roughly speaking, the idea is you’re going to get AI systems that once they can substantially contribute to the process of AI R&D and improving AI systems, you’re going to get this rapid process of recursive self-improvement where every AI system is sort of iteratively involved in building better and better AI systems. We should actually, I think, do a whole separate episode on the intelligence explosion. Because I think reading around about AI, so much of what people are thinking about the future seems to depend on their assessment of like the plausibility of that intelligence explosion concept.
But yeah, so I think you might though think of that as part of that first category of sort of defining AI by its impact in a sense. So AGI by its impact. So there AGI would be, you know, whatever triggers this hypothetical intelligence explosion.
I mean, I think that in addition to what you said, a general problem with defining AGI in terms of the impact of AI, where you’re sort of neutral on what kinds of capabilities might produce that impact, it’s not really forward looking, that kind of definition, right? It’s in a sense, almost by its very nature, going to be backward looking. And it’s not really clear then what we should be searching for or how you would go about measuring AGI solely by looking at the capabilities of the AI systems themselves. So even though I think there is a place for this idea that we need to be thinking seriously about what a world will look like where you’ve got radically transformative AI, merely having this placeholder, to me at least, doesn’t seem that useful as a way of understanding this concept of AGI.
Henry Shevlin: Yeah, I agree. And I think the idea of transformative AI is a useful concept in itself, but I do think it’s worth distinguishing from the kind of more cognitive scientific concept of AGI for a couple of reasons.
The first is that I think you can achieve transformative AGI, sorry, transformative AI, even with quite narrow systems. So there’s this really interesting idea that was very, very central to a lot of AI discourse in the late 2010s called comprehensive AI services. So this is an idea developed by Eric Drexler who said, look, maybe it would be a good idea for safety reasons if rather than trying to build one AI to rule them all, we focus on more narrow domain expert AI systems. So you’ve got an amazing AI scientist, an amazing AI financial analyst, you’ve got an amazing AI writer, but they’re not joined up. They don’t talk to each other at least directly.
And that could be better from a safety perspective, but also pretty much just as useful as AGI. So this is often framed as sort of a choice between two different directions that the future of AI research could go. Part of the problem there is I think LLMs kind of fall between the cracks of AGI and CAIS, as it’s called, Comprehensive AI Services. But insofar as they are a sort of unified system in some sense in terms of their generality, they can do lots and lots of different tasks and they’re not narrow systems. But at the same time, they’re not unified in the sense of being a single psychological agent with memory carried across different instances, capable of coordinating thousands or hundreds of thousands or millions of different conversations towards a single goal. And of course, a lot of the power that LLMs have is their ability increasingly to use various tools rather than sort of having those tools integrated into the systems themselves.
So I think there’s a world in which AI turns out to be transformative that ends up looking a lot more like Eric Drexler’s world. So this is a world without AGI in the sense of, you know, one system to rule them all. Instead, lots of powerful specialized systems, but that still utterly transforms our society and economy. So that’s one reason I think the transformative definition is maybe worth separating out.
Another reason to separate out transformative AI from AGI is something that’s been a big issue in the last year, which is adoption. We could have amazing AI systems or increasingly powerful AI systems, but due to economic or structural factors, they don’t end up at least straight away having the kind of transformative impact that people I think sometimes slightly naively assumed would just happen straight away as soon as you get AGI. So again, you might have, so it’s a sort of a double dissociation. You might have transformative AI that still falls short of AGI because it’s something a bit more like CAIS. Or you might have genuine AGI, but it’s not yet or not immediately transformative because of structural, legal, economic obstacles, things like adoption, to prevent it having the full impact.
Dan Williams: Yeah, and I think that idea that you can’t leap straight from the capabilities of the system to its real world impact is a very important idea in thinking about AI in general. And in fact, we touched on this in our first episode where we looked at the AI as normal technology perspective from Arvind Narayanan and Sayesh Kapoor, where they make a really big deal of this idea that diffusion takes time. There are lots of bottlenecks. There’s going to be lots of risk aversion, need to have all of these other complementary innovations within society in order to actually integrate AI capabilities. I think that’s really important.
Maybe to sort of take a step back. So as I understand where the concept of AGI sort of first comes from, when we separate it from these questions of real world impact and just focus on the capabilities of a system, is if you look at the history of AI, we have lots of very impressive systems, often superhuman, along relatively narrow dimensions, but that could only do some things. So a chess playing system that will destroy the world’s best chess player, but it can’t really do anything else. And even if you just change the rules of chess very slightly, the systems are so brittle that suddenly they’ll lose all of their capabilities.
And one thought was, well, that’s one kind of intelligence, a kind of narrow intelligence, which these AI systems that we were building possessed. But in principle, there could be a kind of intelligence where it’s incredibly flexible and open-ended in terms of the kinds of tasks, the kinds of goals that the system could achieve. And then I take it a question people are gonna have is, okay, why should we expect such a system is even possible?
And a thought many people have is, well, we have human beings, right? And human beings are a kind of existence proof for a certain kind of highly general, flexible, open-ended intelligence, in as much as human beings can become poets, scientists, engineers, dancers, we can play an open-ended set of possible games, and so on and so forth. So the idea is there’s gonna be a kind of conceptual contrast between narrow intelligence and general intelligence. And in a way of addressing skepticism about the possibility of general intelligence, people can always say, human beings have this kind of generality in terms of the sorts of things that we can do.
And I take it that’s partly why so much of the AGI discourse gets translated into human level AI discourse because human beings are supposed to be this kind of existence proof for the kind of intelligence that we’re thinking about. I’m really torn here because I think clearly on the one hand it is true that human beings have a kind of flexible open-ended intelligence that can be combined with an open-ended set of goals and we can perform a variety of different tasks. On the other hand I do really worry about this concept of human level AI, it feels a little bit incoherent to me, like we’re dealing with a kind of great chain of being where there’s this single quantity of intelligence and human beings were on a certain level and we just need to get to that level. That feels a bit confused and sort of dubious to me.
I also think, and actually maybe this is an area where we disagree, ultimately it’s not obvious to me that you’re going to be able to build systems that can do everything that human beings can do that work radically differently from human beings and are subject to a totally different kind of design process in terms of the learning mechanisms by which they arise. I think that idea is coherent, but I think this concept of AGI is basically saying we’re going to get systems that can do everything that human beings can do. They’ve got the kind of flexible, open-ended intelligence, but they’re not going to work anything like how human beings work.
I feel like that idea doesn’t get enough scrutiny in discourse about AI. What do you think?
Henry Shevlin: So loads of juicy threads there. Just a couple of quick historical notes. So the idea of generality as a feature of AI systems was really popularized by John McCarthy all the way back in the sort 60s and 70s, one of the founding figures of modern machine learning and AI. And then I think AGI or the concept of general intelligence as a central notion for frontier model development is sort of popularized and refined a bit by Shane Legge and Marcus Hutter in the early 2000s. So they give this famous definition of general intelligence as the ability to achieve goals across a wide range of environments.
And if we’re going to sort of do any useful sort of scientific analysis, I think, with the concepts in this vicinity, I think the idea of generality as a sort of continuous dimension is more useful and interesting than the concept of AGI per se. I think the AGI sounds like there’s a definite finish line for model development, which I think is probably unlikely for reasons maybe we’ll get onto, but spoiler alert, I think it has to do with the jagged frontier and the jagged nature of AI development. But on the other hand, the idea of generality seems like a really legitimate scientific category, right? Be able to measure, you know, obviously operationalizing these terms is always a bit tricky, but the idea that we can measure the ability of systems to perform well across different domains, that seems like something that is measurable and is meaningful. And I think that’s an area where we’ve seen astonishing progress in very, very recent history.
So back in, I think it was 2019, I wrote a paper with Karina Vould, Matt Crosby and Marta Helena called The Limits of Machine Intelligence, where we were comparing contemporary frontier AI systems somewhat negatively with capabilities, not just of humans, but of non-human animals. In that paper, we draw heavily on biology and just talk about the wide range of things that honeybees can do that birds can do, how they are not specialized intelligences and comparing them with things like AlphaGo or AlphaFold, which are, as you sort of suggested, really, really powerful systems, but operating in very, very narrow domains.
Now, since then, and somewhat, I think, to the surprise of me and others, large language models have shown that in some ways it is possible to build really quite robust systems, systems with a very high degree of generality across a lot of cognitive tasks. And I think that this has sort of dawned quite slowly. I think as recently as sort of just like the launch version of ChatGPT, which was running on 3.5, you still ran into a lot of the kind of familiar problems that you’d run into with sort of previous systems that you alluded to. You change the rules of chess slightly and you get sort of inelegant failures. And I think you could see that already with things like ChatGPT, the launch version would often make non sequiturs. It was easy to confuse. Fairly trivial to get it to hallucinate. And across all those metrics, these systems have been getting more and more reliable.
Early ChatGPT was terrible at mathematics, for example. Contemporary ChatGPT or contemporary LLMs in general can do fantastic mathematics. We’ve had admittedly specialized fine-tuned models, but still LLMs at core that are now winning International Math Olympiad goals. So I think maybe one way to push back against your idea that generality, or at least your hypothesis that maybe generality, high levels of generality, are only achievable in something like a human package. Well, I think the trend line suggests that we are moving rapidly towards more and more general systems in a distinctly unhuman-like package in the form of LLMs.
Dan Williams: Completely agree. And this is, I think, the kind of strongest argument for the alternative view. I mean, just to kind of reconstruct my somewhat garbled reasoning, my thought was something like, we talk about AGI, and often the existence proof that there’s such a thing as AGI is the fact that we’ve got human beings. And I think so much of the discourse about why AGI will be transformative is the idea that these systems will be able to do everything that human beings can do, maybe just in the cognitive, intellectual domains.
And my thought was, well, fair enough, but we’re not building systems that work anything like the human mind, anything like how the human mind works. So there’s a kind of assumption here, sort of bundled with this AGI concept in terms of the way that it gets used, which is we’re going to build systems or we can build systems, maybe we are on track to build systems that can do everything that human beings can do in a way that this concept of AGI sort of captures, but that work nothing like human beings. And I don’t think it’s obvious that that assumption is true. A priori, certainly being a physicalist, being a functionalist doesn’t commit you to the truth of that. So the question is, why should we believe it?
And I think a very good response is look at what’s happening in AI over the past few years. Maybe a kind of skepticism made sense in 2020. But now, just given the realities of how much AI progress there’s been, especially when it comes to the generality of these LLM-based systems, that skepticism is difficult to maintain. I think that’s fair. I definitely think that the progress that we’ve seen in AI and the fact that clearly a significant aspect of this progress is the generality of problem solving ability with these systems. I think that does put pressure on the kind of skepticism that I was raising.
I do wonder how much pressure. Like suppose someone just wants to say, okay, you’ve made a certain kind of progress. We can characterize that in terms of generality. But of course, the people who are really bullish on AI progress, they’re not just claiming that these systems are very competent and general as we find them today, they’re claiming that we’re going to have drop-in workers that can substitute for human labor across different areas of the economy. Why should we extrapolate from progress that we’ve seen over the past four years and think that that’s going to get us to the full suite of capabilities that we associate with human intelligence? We’re kind of skipping ahead here to get to questions about benchmarks and progress and so on. But I think it’s an interesting question. What are your thoughts about that?
Henry Shevlin: Yeah, I think you’ve characterized the debate really well. And I think it was a really plausible hypothesis, even a couple of years ago, that, you know, to use the meme, the phrase that has rapidly become a Twitter meme, you know, deep learning is hitting a wall, LLMs are going to hit a wall, that it was like a really viable empirical hypothesis that we’d find out that there’s only so far you can go with these very unhuman-like architectures. Okay, maybe we find out that you can use them to generate high quality code and do basic composition and translation. But there is some sort of task set T where no matter how big we build the models, they’re just no good. Maybe that would be social cognition or causal reasoning or scientific reasoning.
And yet every candidate domain pretty much has fallen. So I think that doesn’t mean that we won’t find some candidate domains where it turns out these systems just, where just scaling these systems up won’t lead us to greater progress, but we haven’t found them yet.
I think probably the most interesting one that I’m watching at the moment is agency. Some of, I think, I’m not sure if we’ve discussed it before, but things like Anthropic’s experiment with Claudius getting Claude to run vending machines at Anthropic’s offices and failing abysmally any kind of like long term structured planning task that involves interacting with different human agents, some of which might have slightly malicious motives, you know, people trying to get discounts from the vending machine. It’s very funny. We can probably drop a link to the study in the blog. But that was an area where it looks like we really are struggling to build systems that can do something like sustain human agency. But even there, we’re seeing rapid progress. And it’s not clear to me that we’re immediately hitting any sort of brick walls.
So that said, it is entirely possible. And I’d also just emphasize again that I think this is very much an empirical question. I think, again, it was a really plausible hypothesis a few years ago to think that simply training on language alone wouldn’t be able to get you anything like cognition. I think there’s a natural vision of how cognition works where, in the human case, language sort of sits at the top of the pyramid. And then you’ve got layers underneath of things like sensorimotor cognition, motor skills, spatial reasoning, social reasoning and so forth. And language is just the capstone. And if you try and build that capstone without the supporting layers, sure, you might be able to do some clever stuff, but it’s never gonna give you real intelligence.
And I think the discovery that at least that doesn’t seem to be the case from what we’ve seen so far is from just a general cognitive scientific point of view, probably the most astonishing discovery in cog-sci in several decades, I think.
Dan Williams: Can I just quickly interrupt, Henry, because I really want to make sure that I’m understanding what you’re saying. So the last thing you said was you might have a model of kind of agency and intelligence where you need to get the sensorimotor stuff, the kind of embodiment, the being in the world. Is that a Heidegger phrase? I’ve no idea what he meant by that. But that kind of stuff, you need to get that basic sensorimotor stuff, lots of the stuff that we share with other animals, right, first before you can get these more kind of cerebral intellectual tasks like being amazing at software engineering and coding and mathematics and language and so on. And your thought was that was an interesting hypothesis. Actually, what we found with AI in the past few years is it’s not true. Actually, you can get all of that really kind of cerebral, highly intellectualized, those sorts of capabilities without that other stuff.
But couldn’t someone say, well, that sort of cuts both ways in a sense. So we’ve talked about this previously, but there’s this famous, you know, Moravec’s paradox, you know, things that we find easy are hard. Things that we find hard are relatively easy. And that what we found with AI progress over the past several years is, yeah, these systems have got really good with these kind of, we might think of them as evolutionarily recent capabilities that human beings have these very abstract cerebral intellectualized stuff to do with manipulating text and so on. But real, kind of, the significant challenge when it comes to intelligence isn’t that stuff, it’s that sort of basic sensorimotor coordination, these much lower level abilities that we share with other animals. And so far we haven’t seen much progress on those things. And therefore we shouldn’t actually be so bullish on the progress that we’ve seen with these AI systems over the past few years.
Henry Shevlin: Yeah, again, really interesting. I think Moravec’s paradox is looking a lot shakier than it used to. So for it, I mean, one example of something that was sometimes cited as sort of an instance of Moravec’s paradox was image recognition. Image recognition was famously incredibly, incredibly hard, correctly categorizing the kind of things that were in a presented image. And then around 2012, things like AlexNet was one of the early deep learning systems that started to radically tear away these benchmarks and start to dramatically improve on previous generations of performance. And I think it’s fair to say that image categorization is basically a solved problem now.
And I think in quite a few Moravec-type domains, we’ve seen very, very rapid progress. So another Moravec-ish domain is things like understanding conversational implicature or subtle things that people might mean. So conversational implicature, a technical philosophical term, but huge amounts of human language or huge amounts of human communication rely on things like theory of mind and shared context. So if I say, what do you think of that? Whether that is referring back to something I said five minutes ago, being able to figure out what I’m referring to, that’s a very Moravec-style skill that relies on a lot of contextual knowledge. But in these kind of domains, AI just does brilliantly nowadays. AI is very good at conversational pragmatics or conversational implicature, very good at image recognition.
So Moravec’s paradox is no longer, it’s no longer clear that it holds or it holds in a much more uneven and jagged way. It’s not the case that sort of everything that’s easy for a two year old is hard for AI and vice versa. So I think that’s one of the ways in which I would push back.
Regarding the broader question, sure you’ve built sophisticated language models. That doesn’t mean that these systems will then be able to do the fancy sensory motor stuff. I agree. I think that’s absolutely right. So it may not be the case that LLMs five years from now are any better. Well, I think they will be at least a little bit better at the kind of sensory motor stuff as we’re seeing from the increased integration of sort of LLMs into robotic architectures and so forth. But yeah, I think it’s definitely possible that we found a different way to build high-level intellectual capabilities that doesn’t translate to sensorimotor capabilities.
But the other thing I would flag here, and maybe this slightly undermines my own point from earlier on, is that contemporary LLMs are radically different beasts from LLMs three years ago. Contemporary LLMs interpret live video. They interact with the world via querying web results. They can access APIs. They can use tools. They are in a kind of dynamic relationship with the world, albeit one that’s a little bit different from ours. You can ask ChatGPT, is this bar open on a Friday? And it’ll say, yes, I think it is. And say, can you check that? And it’ll come back and say, I’m wrong. Sorry. Yes, they’ve just recently changed their opening hours. They’re now closed on a Friday. I think that is almost a form of sensory motor grounding, you know, in obviously a different package. But so I think contemporary LLMs are, they’re not just sort of these ossified monoliths trained on a bunch of text and then frozen in time forever. They are in some ways closer in, at least at a very abstract architectural level to the kind of dynamic, quasi-embodied systems that we are.
Dan Williams: Interesting. I’m not so sure that Moravec’s paradox has been challenged to the extent that you’re suggesting. I mean, we don’t, we don’t have robotics, right? It’s nowhere near as advanced as these LLM-based systems.
Henry Shevlin: Well, hold on, hold on. Just on that point, what do you think of driverless cars as a counter example here? Because driverless cars were another one of these things that where people often used the failures of driverless cars in the 2010s as an example of Moravec’s paradox in action. They said, these people, actually things like driverless cars are to be far harder than people realize because it involves this whole complex suite of sensory motor capabilities. But now, the safety record of Waymo in the Bay Area exceeds that of human drivers.
Dan Williams: Yeah, very, very good point. I do think though, some degree of goalposts shifting and realizing that certain things we thought would be very hard and much easier than we thought can kind of be legitimate in this context because our intuitions are not particularly reliable when it comes to tracking what really matters about intelligence.
So if you go back to the seventies and eighties, all of these people, even those who thought that embodiment was really central to intelligence, they would say things like, well, you’ll never get an AI system that can beat a human being at chess because that’s going to tap into all of this constellation of abilities, which are, you know, connected to our embodiment and so on. And then obviously we know what happened, but I think part of that is we’re just learning with every kind of development with these AI systems that there’s much more to intelligence than we thought. So yes, we do have self-driving cars, but we don’t have functional robotics of the sort that we can integrate into our lives, suggesting that self-driving cars as impressive as that kind of technology is, is not really a proxy for the kind of full suite of sensory motor abilities that we care about when it comes to animals’ interactions within the world.
I think we’ve also so far been thinking of Moravec’s paradox in terms of this contrast between the highly cerebral intellectual domains, kind of symbolic, often explicitly text-based and basic sensory motor control. But I think there are things like continual learning, right? The capacity of animals, very, very young children, a perfect example of this, to be constantly learning from their environments. And in a way, I think that’s one of these things which state of the art AI today hasn’t cracked. I mean, you’ve got this kind of pre-training phase where it’s next token prediction. Then you’ve got post-training where it’s various sort of reinforcement learning-based learning processes for the most part. But you don’t have kind of continual learning, updating of the model weights as they go through the world from their experience. And that’s not strictly speaking, just a sensorimotor thing. That’s also connected to our sort of higher abilities.
And then also things like, you know, creativity, judgment. We’ve got these words for these concepts. And I think our explicit understanding of them is quite weak. But I do think there’s something to the idea that, you know, ChatGPT, the amount of knowledge this system has is unimaginable relative to what an individual human being has. But individual human beings can do things in the cognitive domain, which is still much more impressive than what systems like ChatGPT or Gemini can do. And again, that’s sort of, it’s not, it’s a kind of competence, it’s a kind of ability, which is not purely sensory motor, but what I think is quite central to how animals in general go through the world, capacity for judgment, for creativity and so on, which again, these systems don’t seem to possess.
And one reason for that might be that they’re just these incredibly weird systems relative to human beings. Their training process is completely different. Their architecture is completely different. And they can do these things that are incredibly impressive, almost unimaginably impressive relative to a few years ago. But there’s a great quote actually from AI podcasting legend, Dwarkesh Patel, which is something like these systems are getting more and more impressive at the rate the short timelines people predict, but more and more useful at the rate the long timelines predict. The thought being, yes, what they can do in terms of our subjective sense of how impressive it is, is amazing. And they’re performing very, very well in terms of these benchmarks. But in terms of their real world utility, actually they’re not having the impact that many people think. And one reason for that might be that they lack many of these kind of amorphous, nebulous capabilities that human beings and indeed to some extent other animals have. Sorry, that was me. I’m not, that was very nebulous and sort of inchoate in terms of their thoughts there, but I’ll be interested to hear what you think.
Henry Shevlin: Well, can I ask, what are some examples of judgment or creativity involving tasks where you think contemporary models clearly fall short of human capabilities? And I’m not denying that there might be such cases, but I’m just curious if there are any ones you have in mind.
Dan Williams: Yeah. Well, for example, I mean, I’m a writer and a researcher. I don’t think AI systems as they exist today, or maybe I should actually, I should rephrase that as commercially released AI systems, because God knows what’s happening privately within these labs. I don’t think they could function as a researcher and as a sort of writer generating novel and interesting opinions, which is the kind of self image that I would like to have. I think they can write bloody well. And I think if you use them as an assistant, it can be incredibly helpful in terms of augmenting and enhancing your abilities. I don’t think we’re at the stage where a ChatGPT could function as a substitute for me, which in a way is strange because it has a knowledge base, which is sort of just so vast relative to my knowledge base or the knowledge base of any other kind of researcher.
So I would imagine if you took my abilities, limited as they are, but combined them with this kind of almost godlike knowledge base of the sort of these systems have, you would get really, really kind of impressive research outputs. But you just don’t see that when it comes to these state of the art AI systems. Am I missing something? Do you disagree?
Henry Shevlin: Well, I think one thing that’s worth mentioning is I think it might be a little misleading to compare you who are, you know, I think, I hope you won’t mind me saying an elite sort of knowledge worker, right, with in thinking about sort of your ability to do original composition, original essays, original analysis. Yeah, I think you still have an edge. But I think we’re well past the point where sort of the median undergraduate essay, I mean, the ChatGPT in its current form can produce far better essays than the median undergraduate essay. I think at this stage, in some domains, it can produce far better essays than the median grad student 5,000 word essay.
And so I think there’s a little bit of a tension there if you’re saying, humans in general have this special sauce that lets us do things that AI systems can’t, when in fact, already AI systems in the kind of domains you mentioned already do vastly better than the very large majority of humans within these tasks.
Dan Williams: Yeah, I think I’m more open to the possibility that they’re doing something very, very weird, incredibly impressive, that does seem to outcompete human beings across specific tasks, but they do in fact lack many properties and capabilities that human beings have such that they couldn’t substitute for them even when it comes to purely intellectual tasks. I do realize though, there is the possibility of a significant amount of copium, self-serving cope in terms of this. And there’s something unsatisfying about it as well, in as much as I think you’re right to, you’re really right to push back. And also I would say, I wouldn’t have predicted the progress of the sort that we’re seeing back in 2020. And I think I probably haven’t fully updated to the extent that a rational individual should have done given the kind of progress that we’ve seen.
But let’s just quickly sort of, let’s return to this, but I’m aware of the fact that we got derailed by a really interesting conversation there. And just take a sort of detour through, we’ve touched on this introductory stuff about kind of AGI. So, you know, how you might understand the concept in terms of transformative impact, in terms of human level AI, in terms of more abstract sort of functional specification of capabilities. Maybe we can just spend a little bit thinking about the skeptical arguments concerning this concept of AGI. So like people like Jan LeCun or Alison Gopnik that I mentioned at the beginning, just saying the concept makes no sense at all and there’s no such thing as general intelligence.
I take it, I mean, one argument you often find here is that human beings are supposed to be the existence proof for the AGI. Here is a, you know, complex information processing system that has the kind of set of capabilities that people that talk about AGI are interested in. But the thought goes, well, human intelligence is not general. The human brain is this integrated mosaic of very specialized abilities that correspond to the kinds of problems we confronted in our kind of evolutionary past.
Sometimes this is cashed out in terms of like massive modularity to get a bit nerdy in terms of the cognitive science debate. And I think people into that kind of perspective, they think there’s something problematic with the concept of AGI because it seems to assume that intelligence is this one generic problem solving ability when in fact human intelligence, which is supposed to be our only existence proof of AGI, doesn’t take that form. It’s this set of special purpose modules for different tasks, which might be nicely integrated in the case of the human brain, but don’t involve just general purpose sort of learning mechanisms. What’s your thought about that kind of critique or that kind of worry?
Henry Shevlin: Yeah, so I’m pretty sympathetic to massive modularity in the human case. I think if you are sympathetic to massive modularity in the human case, that just seems like one way of interpreting that is to say that AGI can operate across, or that general intelligence can operate across highly modular architectures. If what we’re thinking about when we’re thinking about general intelligence is something ultimately grounded in the ability to perform cognitive tasks, right? Does it matter whether that’s achieved purely via a relatively narrow bundle of cells all in your prefrontal cortex or using working memory, or if it’s a bunch of different sort of cognitive sub-modules working together.
So yeah, I think if you accept the massive modularity as a thesis in humans, then why not just say, okay, so maybe the way we get to artificial general intelligence is through a similarly massively modular system. And you can already see hints of this in the way that in the increasing tool use by AI systems.
And it may be that, and this sort of goes back to our discussions about CAIS versus AGI, that the first kind of true AGI systems, I’m skeptical we’ll ever have like a clear, we’ve built AGI moment. But maybe the first systems that sort of get most people would agree are AGI systems might similarly have a relatively modular architecture, maybe with sort of a central coordinator powered by an LLM coupled with a dedicated mathematics engine, coupled with dedicated deep reinforcement learning agents for doing various kinds of scientific work, coupled with, you know, maybe sensory motor systems embedded in drones for doing that kind of thing. I think that would still be AGI, at least in the sense that it’s sort of relevant and interesting.
Dan Williams: Yeah, that’s a very, I think that’s a very good response. I mean, is the worry then that these people have that actually, if you look at AI as it exists today, most of what’s powering it is very general purpose learning mechanisms that doesn’t really look like what you’ve got in the human case. So maybe we should be skeptical that you’re going to get to human-like capabilities via this architecture. But I think your point that actually there’s a lot more kind of modularity here than you might think if you just look at the base model precisely because of this interface with all of these mechanisms. I think that’s important.
I wonder if there’s another kind of thing in the background here, which is skepticism about the way in which AGI often gets talked about where it’s like, we’re gonna build AGI and it’s gonna have these almost superhuman capabilities across all of these different domains. And maybe some people think, well, if you look at human beings, existence proof for this concept of AGI, you don’t find anything like that. You find that we’re very good at some things, we’re not so good at other things. So maybe the thought would be once you’ve paid attention to how human intelligence and maybe more broadly kind of animal intelligence works, very kind of specialized, very modular, that should make you a bit more skeptical maybe about some of the claims about the capabilities of super intelligent AGI in the future. What do you think of that kind of argument?
Henry Shevlin: Yeah, I mean, I think it’s definitely worth stressing how sort of distributed our civilizational capabilities are across different humans, right? I think most humans are not fully general in their intelligence. Some people are great at mathematics, some people are great at coding, some people are great at languages. But we’re able to achieve remarkable things at the civilizational level or at the cultural level because of cooperation across different kinds of specialists within our massive population.
But again, I don’t see why a model like that couldn’t apply to AI systems. Maybe that’s across millions of different instances with different fine tuning to different tasks. So yeah, I think the lack of generality in individual humans is compensated for at the population level. And I don’t see why a similar kind of distributed architecture couldn’t apply to relatively near future AI systems in the kind of modular way I’ve been describing.
There’s an idea here worth bringing back that I touched on earlier on, which is the jagged nature of current AI systems. So for anyone who’s not familiar with this, roughly the idea is if you think about sort of a spider diagram or a radar chart, as it’s sometimes called, where you sort of think about different dimensions of intelligence and sort of map human performance on this, you know, we’ve got sort of spatial reasoning, mathematical reasoning. And let’s just say for the purposes of argument that humans are pretty well rounded across this domain.
AI systems, current AI systems are really, really superhuman already at some tasks, well below human performance on others, around human performance on some. I think this is a really striking observation and a really important observation for understanding trends in current AI. And also explains a lot about the point you made earlier about why these things are maybe less useful than you might have expected.
And, you know, I’ll happily say on the record here that I was far more optimistic about the near term economic impacts of things like ChatGPT, then turned out to be actually correct. If you’d asked me, well, I think I was saying back in November, 2022 on Twitter and places that this is going to revolutionize the economy in the next few years. I still think it is going to revolutionize the economy, but it’s been a lot slower than expected. And I think jaggedness is a big part of the reason, adoption is another.
But just to sort of go into this a little bit more detail, when we think about what an individual human job involves. It involves a huge range of tasks. It’s not one task for the most part. They’re bundled tasks. Current AI systems are really good at some and bad at others, which makes the idea of the drop-in agent-employee model currently non-viable because there are enough tasks within human workflow that AI is really bad at to mean that’s just not applicable.
So a couple of things you might say, how we’re to get around this problem. One is that just rely on these systems getting better and that jaggedness, if not smoothing out, then to sort of the sheer level of the abilities expanding sufficiently that, you know, even if AI systems are still vastly superhuman in some domains and only human level in others, they’ll be good enough across the board that they will be able to function as drop-in agents.
Another interesting idea though is that we will just redesign task flows. We will do some unbundling of tasks in roles such that we create sort of roles that AIs can be dropped in on quite safely. I think a nice useful analogy here, I was talking about this on Twitter not long ago, is if you look at mechanization in agriculture, right, it’s not the case that mechanization in agriculture proceeded through creating robot farmers. It involved instead changing task flows such that relatively simple machines could take over very sort of labor-intensive tasks from humans and changing the kind of things that the average human farmer does.
I think that might be a better model for thinking about at least near-term AI impacts on employment, where it’s a matter of redesigning task flows and value chains such that there are, we do create these niches where you can drop in these AI agents to take on huge important parts of the value chain without necessarily replacing humans one-for-one on the kind of jobs that humans currently have.
Dan Williams: Yeah, that’s really interesting. And I think it’s a very insightful observation. I mean, I would say though, when we’re thinking about what people do in their jobs, it’s not like, you know, there’s a set of tasks that are separate from each other and, you know, AI can do 40% of them or soon it’s going to be able to do 60% of them. The tasks are integrated with each other in an incredibly complex way, such that we might be able to delegate some of these individual tasks to an AI system. But if I think about my job as an academic at a university, it’s not like I can say, my job consists of 142 tasks and here they are. It’s a much more integrated kind of unified set of responsibilities and obligations.
So I think if we’re thinking about not just delegating some tasks to AI systems and adjusting how the workflow is structured and adjusting the structure of organizations, but thinking about radical forms of automation. At the moment, I think that we’re very far from that precisely because I don’t think even as impressive as these AI systems have been, they’re capable of that kind of really kind of long time horizon integrated, like multimodal task performance of the sort that most human beings perform.
And that actually gets us nicely onto something we’ve already touched on, but I think we should think about and talk about as a kind of separate topic, which is measuring progress in AI. So lots of this is framed in terms of progress towards AGI. But I guess you can just think of it in terms of the progress and the capabilities of these systems in general.
So I think there are kind of three overarching ways in which we do this, again, to draw another distinction between three different categories. There’s the kind of subjective, how impressive is this? It’s not completely without value, but I think it is very unreliable for various reasons. There are the sort of set of benchmarks, formal benchmarks that are used to evaluate model performance. And then there is actual kind of real world deployment. So something like what percentage, what fraction of work in the economy is done by automated AI systems or something like that.
If you’re thinking about those three categories, I take it that the quote that I paraphrased from Dwarkesh, where these models are getting more impressive at the rate that the short timelines predict and they’re kind of more useful at the rate that the long timelines people predict. That’s drawing a distinction between two different ways in which you can evaluate these systems. There’s the kind of how subjectively impressive do we find them? And maybe that’s also connected to benchmark performance, where as you’re saying, they’re just getting better and better at a seemingly just ever increasing set of tasks. It’s juxtaposing that with like real world utility. I think that’s complicated, as you said, by the fact that real world deployment is not a simple function of capability, is also going to be shaped by all sorts of other things.
But how are you thinking about this, about measuring AI progress, how to use that to forecast AI progress?
Henry Shevlin: Yeah, I think that’s a fantastic tripartite way of splitting it up. So just a couple of quick comments on sort of the tripartite division. I think as we saw from the launch of GPT-5 last year, how underwhelmed most people were by GPT-5. And I think that that was a fascinating sort sociological episode in itself, particularly because if you look at it purely in terms of benchmarks, the line kept going up and continues to keep going up across most of the things we know how to measure.
There is no evidence of a slowdown in AI capabilities, at least in terms of evals and benchmarks. And yet people were a lot less impressed by GPT-5 than previous models. I think there are a few little just interesting reasons for that. A very basic one is just that cadence of release has massively increased. So it’s no longer the case that we’re waiting a year and a half between model releases with no releases at all in between. Now we get updates pushed every couple of months.
So there are going to be fewer wow moments. I think that sort of partly explains why maybe people were a little bit underwhelmed by GPT-5. Another fact is that I think, another sort of constraint on how impressed people are is that I think models are already good enough at most of the kind of tasks that most people use them for such that new releases don’t radically change people’s affordances with the system.
I mean, I think there are occasional specific domains in which they do. So just to give one example of, I think, a major transition, as it were, in capabilities. I think the release of NanoBanana, Gemini’s integrated image model, dramatically changed what you could do with images and data, specifically because NanoBanana is very good at threading text data and sort of semantic content with images. So for example, with NanoBanana, you can have a long conversation and say, now create an infographic or a mind map of the conversation we’ve just had. NanoBanana from day of release could do that in a way that every other previous image model would fail abysmally at. So there are these kind of like sudden, wow, here’s something new that I could do that I couldn’t do before. But in terms of just sort of general performance of language models, I think they are good enough for most purposes that there are fewer wow moments there.
I think more broadly regarding which of these three ways of measuring AI progress are most relevant and important. I think it depends on the domain. So I think there are isolated domains as measured by particular benchmarks, like some of the mathematics benchmarks, where those benchmarks do have immediate significance, right? If we are using AI models to solve near future, outstanding major problems in mathematics, right? Then I think benchmarks might be getting us pretty close to measuring the underlying criterion that we’re interested in.
Ultimately though, I think it’s the economic impacts that are most pressing and most exciting and most scary. But as you said, there’s so much more to those than just the raw capabilities of the system.
Dan Williams: Yeah. I do think though, I mean, now I’m just becoming a Dwarkesh fanboy, but just to observe another point that he made in this blog post, and we can link to this because I think it was a very interesting one. He says, people who make this point about the difference between, you know, the raw capabilities of the system and the rate of diffusion, or in other words, say that you can’t evaluate the capabilities of the system merely by looking at the degree to which it’s integrated into people’s workflow because that’s going to be slowed down by all sorts of factors. He says that’s basically a cope on the grounds that if we really had AGI, it would integrate incredibly quickly.
And I think an analogy he uses, if you think about, you know, immigrants integrate into the economy very, very quickly because they’ve got this wonderful, flexible, like general purpose intelligence that human beings have. And he says, well, if you really are imagining an AGI of the sort that people like Sam Altman and so on were forecasting, then you wouldn’t have all of this friction when it comes to integrating these systems into an organization’s workflow because they will be able to do everything that a human being can do just better. So it wouldn’t be any more difficult than integrating a human being into it.
It’s an interesting argument. I don’t know whether I’m sort of fully persuaded by it. Before we move on, did you want to respond to it?
Henry Shevlin: Yeah, I think it’s a fantastic argument. I think actually one useful, I think as we move towards greater and greater degrees of generality, the kind of existing structural constraints and the problems imposed by the jaggedness of models are going to become less pronounced. So I think that is a useful way to sort of measure progress towards AGI is thinking about the degree to which systems are capable of overcoming sort of external constraints, external limitations.
So, you know, for example, something as simple as a model being assigned a task, realizing it doesn’t have the internal resources to solve that task, identifying tools it could use so that it could solve that task, and then using those tools. I think that is the kind of behavior. I think we see some of that already with stuff like Claude Code and its current form. But I think that is a good way to think about what progress towards AGI looks like, overcoming the kind of structural constraints that may not be pure limitations of the model, but it has something important about the model if it can work in indirect ways to overcome them.
Dan Williams: Yeah, and I think it’s also interesting because once you start thinking in that way, it really does put pressure on this sort of AI as normal technology perspective that says, look, if you look at the history of technology, the process by which these technologies diffuse throughout the economy and throughout society more broadly, it takes a long time for all sorts of reasons. You might think, okay, but you can’t look at the history of previous technology because something like AGI would be a radically kind of sui generis technology precisely because it will be very easy to very quickly integrate into people’s workflow and into the sorts of things that companies and so on are doing.
Maybe just on this issue of these different ways of evaluating AI progress, before we move on, we could touch on two different things. The first is, so probably the most influential benchmark at the moment is this METR or MATR. I don’t know exactly how you pronounce it. Model Evaluation and Threat Research graph. I think to be honest, we’d have to do a whole separate episode on this where we really get into the weeds because I think the methodology and everything is very, very complicated. But basically, as I understand what METR are doing with this metric, which is basically a time horizon metric, is it saying, look, lots of other benchmarks, they’re evaluating AI’s ability to perform a set of say abstract cognitive intellectual tasks. But what we should really care about or at least one thing we should really care about if we’re interested in these things like agency and the ability to master context and this sort of constellation of abilities that seem to go along with agency is sort of how long it takes a human being who’s a professional to perform a task.
And to the extent that AI systems are getting better and better at performing tasks that would take human beings a very long time to do, that’s telling you something really kind of important about the capabilities of these systems and how fast they’re progressing. And as I understand their metric, basically what they’re saying is there’s been a kind of exponential growth such that the task length of tasks that AI systems can perform is doubling something like between every three or every seven months, where task length there is specified by how long it would take a person to perform that task.
So there’s a very nice quote from Roon, who’s a popular social media AI commentator. He says, the METR graph has become a load bearing institution on which our global stock markets depend. And the thought there is many people are looking at this graph and they’re seeing line go up. They’re seeing this rapid progress. They’re extrapolating that into the future. And that’s why there’s so much optimism about the capabilities of these systems and how they’re likely to develop into the future. That was my current understanding of this graph and the metric that it’s using. Do you have any thoughts about this evaluation?
Henry Shevlin: No, I think you did nothing, no major notes. I think you did a great job of describing it. So just a flag that sort of the METR time horizons task is specifically focused on software engineering tasks. So that is a slightly narrower set of tasks, but it’s one that obviously has massive economic value and is also potentially relevant if we’re thinking about any kind of recursive elements in AI development, you know, software engineering tasks are relevant to building AI systems. So to the extent that we’re finding massive time-saving improvements through the use of AI tools, that might be expected itself to accelerate AI development. So that’s another reason I think that this is so important, maybe less so for the stock markets than sort of the more kind of future-oriented predictions about where AI capabilities are going to go from here.
But of course, absolutely. This is probably the most interesting benchmark to watch when thinking about real world impacts of AI. Software engineers are a very, very expensive group of people to employ. And to the extent that AIs function as massive time savers in those tasks and can do more and more complex workflows within these tasks, that has massive real world impact and significance.
Dan Williams: Yeah, and I think this point about overwhelmingly the tasks are kind of software engineering tasks. So, you know, a software engineering task that might take a human being six hours to complete in and of itself, you know, that’s going to limit the generalizability of this metric because you might think lots of tasks within the world just don’t have the structure of software engineering tasks.
But I think there’s also a sort of, I mean, there are also all sorts of methodological questions about how they’re calculating this and so on. And like I say, I think we should do a separate episode where we sort of dig into this in detail. But I think there’s this other issue which is just to do with benchmarks as a whole. In order for us to be able to have graphs like this, we need tasks in which basically there’s a correct answer or a correct output.
I take it one worry here is just the kind of classic Goodhart’s. Is it Goodhart’s law? You know, when a measure becomes a target, it ceases to be a good measure. So a risk that with any given benchmark, we’re getting systems that are getting better and better at doing well on the test in ways that don’t necessarily correlate with the kinds of things that we really care about.
But I think there’s also another worry where even if you set that aside, the worry would be something like, okay, by the very nature of these benchmarks, where there’s a kind of clearly defined correct answer or output, you’re not tapping into the kinds of things that really matter to a lot of human intelligence, where it’s not a simple issue of here’s the finish line or here’s a clearly defined correct answer or correct output. And I mean, how far do you think that skepticism can go? Like if someone says, look, there’s a possibility here that even though we’re seeing rapid progress when it comes to these benchmarks, including this sort of time horizon benchmarks, which seems like it should be really informative. Nevertheless, it’s just not really telling us anything interesting about the sort of broader set of competencies that matter for real world deployment. Like how much skepticism do you think is tenable when it comes to the gap between benchmarks and sort of the capabilities that we really care about.
Henry Shevlin: Yeah, I think it’s a persistent worry all across not just AI research or even ML in the broader sense, but psychology. Criterion problems is sometimes called, shot all over the place. We have a dozen different ways of measuring creativity, which have minimal predictive validity for one another. As soon as you operationalize a really interesting target, you immediately lose many of the features that make it interesting in the first place. So I think it is an absolute legitimate worry.
That said, I think that I should be able to do better than this, just anecdotally, I think we are seeing models become just generally more useful. If they were improving in benchmarks, but that wasn’t translating into actual real world utility on different tasks, that would be a real red flag.
I can speak about your experience, but my experience is that basically every successive model release is at least somewhat better. I can do some new things with it. And that’s why I think the METR Time Horizons benchmark is a valuable one, but why I also think sort of more grounded economic benchmarks, for example, the degree of internal value created by different, by AI usage in different industries, the degree to which industries are successfully implementing AI automation projects and so forth, they’re an absolutely necessary complement because they’re measuring something that still has some criterion problems, like generating economic value, but is much more tangible and less likely to be a mere artifact of our sort of testing of our evaluation framework.
Dan Williams: Yeah, okay, great. Let’s, I think there are two things to kind of finish on. One of them, I think we can be brief because we’ve really kind of already touched on this, but it’s what we should expect the transition to a post-AGI world to be like, however you understand AGI. And the other is predictions for 2026 in terms of how we see these capabilities.
But first, just want to give you an opportunity to have a take on Claude Code. So, I’m sure you’ve also seen a lot of commentary, a lot of buzz, a lot of discourse to the effect that Claude Code, and just for those who aren’t really in the weeds in AI, Anthropic is a frontier cutting edge AI company. They’ve got a model called Claude. And as part of that, they’ve got Claude Code, which is primarily used for sort of software engineers and coders, but apparently it has much broader application.
I should say I haven’t used Claude Code. I do use Claude all of the time, which I think is incredibly impressive. I haven’t used Claude Code. I’m very, very skeptical that it’s AGI or if it is AGI, I think that probably tells us that the concept of AGI can’t do the work that many people have assumed that it can do. Have you got a take on Claude Code before we move on?
Henry Shevlin: So I haven’t played around with it as much as I would have liked. And it is, I think, one of the more daunting models for non-technical people to use. Even installing it, for many people, will be a little bit of an adventure. But particularly speaking to friends in technical whose jobs are primarily technical, the wow factor seems to be huge on the current iteration of Claude Code.
People are talking about how it’s transforming their workflows, enabling them to do a whole suite of tasks they couldn’t have dreamt of doing before. And I do think it is a significant landmark. Yeah, I think it probably is a taste of the kind of capabilities that we’re gonna see over the course of the rest of this decade, where it’s not just people slotting AI to do specific tasks or sub tasks within their own workflows, but being able to delegate whole workflows to Agile Systems.
Dan Williams: Yeah, okay. And that in a way that leads us onto the first of those two points that I wanted to end on, which is how we should think of the transition to a sort of post-AGI world. I mean, I take it there’s a model you sometimes come across where it’s almost like it’s the atom bomb going off in the Manhattan Project. You reach something called AGI and it’s just radically transformative immediately, for various reasons, maybe because of the capacity to take AGI and use it for large scale automation, but also potentially because of the ability of AGI to get involved in the AI R&D process, triggering this kind of intelligence explosion.
I’m really skeptical that that’s the right way to think about it. I think what we’re seeing basically is kind of incremental improvements in the capabilities of these systems when it comes to things like agency, multi-step sort of long time horizon planning, continual learning and so on. I don’t think there’s gonna be like a big bang. I think we’re gonna see this sort of incremental progress where if you compare, you know, one year to three years down the line, it will seem like this huge disparity, but living through it, I think it’s gonna seem very continuous.
And also when it comes to the impact of this kind of technology on the economy for the reasons that we’ve got into. I think there are going to be all sorts of bottlenecks. There’s going to be so much opposition, even when you’ve got capabilities that are very powerful, integrating it into people’s workflow and so on and so forth. So I’m definitely not really expecting a kind of big bang here. And I think people saying that with highly agentic, at least relative to what’s come before AI like Claude Code, you’re seeing kind of baby AGI, I think that might be true to an extent relative to certain understandings of what AGI is, but that just tells us, I think, that AGI isn’t going to be this landmark event. It’s going to be a sort of continuous incremental improvement across lots of different capabilities. So that’s my high level take. Do you have a different take? Do you have to build on that in any way?
Henry Shevlin: Yeah, I think I largely agree with your take that we’re not going to have a sort Trinity test equivalent moment if we’re going to use the analogy of the Manhattan Project, right? There’s not going to be a sudden moment where a lab says we’ve built AGI. Instead, it’ll seem very incremental and continuous to most people, even those who are following what’s happening. And then by the end of this decade, we’ll look back and say, holy shit, how far we’ve come.
And I think I don’t think there’s any reason to think that progress towards the kind of highly general autonomous systems that, or highly general autonomous capabilities that people associate with AGI. I don’t think there’s any reason to think that that progress isn’t continuing the pace. And I do think, to go back to Claude Code, that it is an example of the kinds of really consequential leaps that we’ll see.
So Ethan Mollick today has a, Ethan Mollick has a piece, brand new piece called Claude Code and What Comes Next, where he talks about using Claude Code to generate a passive income and how it creates hundreds of files for him. Having worked autonomously for 74 minutes, it deploys a functional website that could actually take payments. It got various things wrong along the way, but it was a far cry from sort of the Claudius vending machine experiments from earlier this year, earlier last year.
So yeah, I think we’re going to look back at the end of this decade and realize how far we’ve come, but there’s not going to be a single Trinity test style moment. And I think an interesting parallel here is actually with the Turing test. I think a lot of people were expecting the Turing test to be, there would be like a decisive moment where it’s like, wow, computers can now pass the Turing test. I don’t think that would have been a smart thing to think because Turing’s original test is woefully under-specified. He doesn’t sort of give exact time windows and so forth, and there are various constraints you can build in.
But I think at this point now, the Turing test is no longer especially relevant as a measure of AI capabilities. It’s still of interest, but it’s no longer the case that it’s sort of a clear benchmark we’re working towards. We have had multiple instantiations of the Turing test now that show frontier AI systems can fool humans over two or five minute time horizons with basically at 100% success rate, like where humans are chance at guessing whether they’re talking to an AI system or a fellow human.
But it’s like that benchmark slowly faded into the background rather than being a decisive moment. And I think AGI is going to be very similar. By the end of this decade, I do expect that we will have autonomous, agentic AI systems deployed in pretty much every industry. The vast majority of people’s workflows and daily jobs are going to be very, very different. I don’t think by the end of this decade, for what it’s worth, that we’re going to be looking at mass unemployment.
I actually quite like Noah Smith’s, I don’t fully agree with it, but Noah Smith has this sort of piece on how even in an AGI world, we might still have full employment, leveraging this sort of concept of comparative advantage, the idea that there are always going to be things where it’s cheaper to employ or easier to employ a human to do a given task. And I think that’s going to be one of the things that prevents sort of mass technological unemployment. Also things just like compliance and the fact that you’re going to need to have humans on the loop in many tasks, monitoring AI systems to ensure that you’re abiding by regulations. But I do fully expect the increasingly general AI systems about which will be debates around AGI will seem increasingly irrelevant to be ubiquitous by the end of this decade.
Dan Williams: That’s a nice prediction. Yeah. The prediction concerning AGI is that debates around AGI will go the way of debates about the Turing test. Also, just to add to that point you made about the economics of this, I think the comparative advantage point is very interesting and I think it’s very important.
There’s also a kind of obvious thing which sometimes gets missed in questions about automation, which is when we say, you know, AI systems that can, let’s say, outcompete human beings doing what human beings do. Really, the contrast there is outcompete human beings using AI systems. So it’s not like human intelligence is this fixed target, such that we need to build AI systems that can outcompete human beings as they are in 2026. Human intelligence in general depends on all sorts of technological scaffolding and so on. And that makes it a moving target.
I certainly find in my own work, me with AI, so much more productive and effective than me without it. So if you ask, you know, could AI systems beat Dan without using AI? That’s a very different question, I think, than could you design autonomous, you know, flexible, continual learning based AI systems that could outcompete me with access to those systems? And I think that’s also got sort of implications for how we think about the real world impact of all of this.
Henry Shevlin: Yeah, I guess I do want to, before we go into predictions, I just want to add one final sort of coda here, which is I’ve, in that sort of foregoing prediction of what we’ll see this decade is not to be extrapolated outwards. I think there may well be a point probably beyond the point of this decade where things start to get really, really weird, where sort of the degree of the absolute advantage systems have really just fundamentally starts to reshape workflows and value chains in a way where human labor may eventually, and maybe in some point in 2030s, start to struggle to fit in.
So I’m thinking here of this wonderful piece, very far-sighted flash fiction piece by Ted Chiang called Catching Crumbs from the Table, published in Nature Futures a long time ago, about 20 years ago, where he talks about post-human science and the idea that eventually science reaches the point where it can only be done by AI systems. And just because the kinds of theorems, kind of tools being used are just incomprehensible to humans. And he imagines this sort of cottage industry of sort of explainers where humans try and understand, you know, we’ve built this, AI has developed this new alloy that we are completely incapable of understanding in terms of our existing material science, but let’s do our best, right?
So I think it is possible, I think broadly plausible that if we extrapolate far enough outwards that we might start to hit that point. And then I really do think all bets are off. What does human employment in finance look like when you have super intelligent financial managers supervising super intelligent analysts? What is the role for the human there? Is it just going to be that you sit next to your mainframe running 10,000 super intelligent AI finance agents, and if they ever do anything illegal, you get fired. That might be what sort of people’s jobs start to look like at that point. But I think that’s slightly longer time horizons instead. What I see over the course of this decade is not mass unemployment, but definitely radical changes in human workflows.
Dan Williams: I also think like one of the things that at least in my own case, the reason why I find this so difficult to think about is I just don’t know what to make of the intelligence explosion argument. And I feel like the people that are expecting a real sort of discontinuous leap here, at least relative to human timeframes, they’re imagining a process which will be incredibly rapid precisely because of this model about recursive self-improvement.
So Will MacAskill and Fin Moorhouse have a really nice article on the intelligence explosion and how you could basically compress sort of a hundred years of technological progress into, you know, much, much shorter timeframe. And if that kind of analysis of what you might see with this intelligence explosion, as they understand it, is correct, then, you know, my current view that a lot of this is going to be sort of relatively incremental and continuous and there aren’t going to be any sharp breaks might just break down entirely. I feel like I need to get a good grip on what to think about that whole argument. But we’re going to be devoting episodes this year to people that—
Henry Shevlin: Just to tee up one thought here. So on this point, a sort of prologue or a preview of the future episode, I do think for all of the worries, I think in many cases legitimate worries about hype in Silicon Valley, I think that is absolutely a legitimate cause of concern. And the quasi-religious nature, I think Karen Hao was talking about, you know, how there often is a quasi-religious element to some of these predictions. I think that’s absolutely right. I don’t think that’s disqualifying, right? But I think absolutely, if you don’t think there’s something, there’s a religious element in a lot of talk about AI and AGI in particular, then you’re not paying attention. There absolutely is.
So I think that’s true, but there’s also a bias in the opposite direction, normalcy bias, and the meme version of this is nothing ever happens. But I think if you just look at the recent history of our species, we have many discontinuities, whether that’s the Industrial Revolution, the Agricultural Revolution, even biologically, the emergence of multicellular life in the Ediacaran and the Cambrian explosion, right? The history of life on earth and the history of human civilization is full of these sort of major transitions, these relatively rapid discontinuities. So I think the assumption that sort of nothing ever happens or that, you know, things are basically going to tick on as normal is another bias that we need to be wary of.
Dan Williams: Completely agree. Okay, you said predictions end of decade. How about this year? So when we do this conversation at the beginning of 2027, here’s maybe one way of thinking about it, right? Capability predictions is what I think you expect these systems to be able to do by the end of this year that they can’t do now. Economic predictions. And here I think the central question is, is there a financial bubble here which is going to burst and, you know, potentially even initiate another AI winter, you know, a period in which lots of the enthusiasm dissipates and then maybe, you know, political predictions, right? At the moment, I think people are not aware of what’s coming. People generally are pretty hostile towards AI and pretty, pretty fearful of it, but we haven’t really seen kind of coordinated political movements against AI, where that’s a defining issue. Should we expect to have seen that by the end of the year?
Henry Shevlin: Oh, so many tricky questions. You know the famous quote, it’s hard to make predictions, especially about the future. I want to say in some ways it’s almost harder to make predictions about the near term than the long term, right? Insofar as those predictions have to be more fine grained, more falsifiable. You know, we can say, oh yeah, like by 2030 things will be really different. That’s easy, right? Saying like what’s going to be different by end of 2026 in some ways is harder.
So I don’t expect any massive AI bubbles. I think as the industry matures, like we will hear about various midsize AI companies who’ve been selling vaporware, going bankrupt. And I think the usual suspects will sort of call this out and say, aha, you see there’s an AI bubble all along, but I don’t expect it to be an industry wide trend. I don’t even expect it to be a major bubble in sort of frontier LLMs or frontier model developments.
But the other thing I just emphasise on bubbles when people talk about the AI bubble is that AI is rapidly proliferating into a whole bunch of different things. So like driverless cars, for example, which were, you know, often decried as vaporware in the 2010s, they’re absolutely here now. You can take a Waymo in San Francisco and several other cities today. And 2026 is one of the big years for rollouts of driverless cars. You now have Waymos in London, right? And I think there’s some like 30 cities globally introducing driverless car pilots over the course of this year.
So even if it turns out that OpenAI hit a wall or there’s some major scandal or they’re over leveraged, none of which I think is true. But if that did turn out to be the case, it wouldn’t kill AI in the same way that previous sort of AI winters have sort of killed research in AI, not quite across the board, but more broadly. At this point, everything from driverless cars to autonomous weapons systems to AI and medical research, AI and material sciences to AI for a wide range of tasks. I think it’s too diffuse and too plural for any kind of single bubble events to kill the industry as a whole.
But yeah, I also don’t expect any kind of, even a bubble in the domain of language models. So that’s one point.
Another area where I do think in terms of economic impacts, I think those will grow. I think more and more people are gonna be start, gonna be seeing impacts of AI in their workflows. It wouldn’t surprise me if we start to see some big legacy companies really, really struggling because they’re being outcompeted by startups or scale-ups that make better use of AI than them.
I think more and more companies are going to have to face this difficult challenge of, do we go all in on AI at this point, or do we still try and manage a slow transition? So I think it’s going to be a very economically disruptive year ahead. I think part of the reason for that is I think 2026 really will be the year of agents. So a lot of people, I think Sam Altman said 2025 was going to be the year of agents, but I think that was premature. AI, agentic capabilities of AI understood here as sort of their ability to do long-term complex multi-step tasks is only really getting going. But I think particularly as we start to see more deployment of these AI agents that are in turn generating useful training data about what works and what doesn’t, I think we’ll start seeing more and more really valuable AI agentic products over the course of 2026. I think Claude Code is very much a taste of what’s to come. So big economic disruptions by the end of 2026.
And I think to touch on your political point, I think this is going to lead to increasing backlash. A really interesting phenomenon at the moment is that on the right in America, there is a relatively unified or at least superficially unified pro-AI mood. I think a lot of this has to do with the influence of, you know, a lot of big or the alignment of a lot of big tech with the Trump administration, which has its own reasons for being very pro-AI, geopolitical considerations and so forth. But I think one interesting prediction would be that the right wing on the American right in particular, maybe the global right, pro-AI attitude may start to break down.
I think we’re seeing some trends, some signs of this in domains like AI in young people. There’s an increasing number of sort of Republican politicians who are very, very concerned about things like LLM psychosis, about appropriateness of content that minors are accessing, about impact on youth mental health.
And I think we might actually start to see, that’s one of the areas where we might start to see in the US context, some bipartisan consensus emerging on the need for AI regulation. I’d say partly that’s maybe due to things like family values, protecting young people being values that are as central, if not more central to the right than on the left. So it’s inherently bipartisan, but also because the idea of better regulations around protecting young people don’t necessarily interfere with kind of geopolitical applications of AI. You know, strict rules on under 18s using ChatGPT is not going to prevent the US from using AI tools effectively in future military conflicts and so forth. So that’s one political development.
I think at the cultural level, things are just going to get weirder and weirder. So, you know, we’ve done two episodes on social AI. I think 2026 social AI is going to continue to become more and more ubiquitous.
Sadly, I think we will see many more New York Times and legacy media stories about LLM psychosis, LLM exacerbated or triggered or implicated suicides. I think we’re going to continue to see deep entanglements, deep relationships between humans and AI systems become more and more common. And maybe an outside prediction, I do think the kind of AI welfare, robot rights movement is going to continue to gather steam. Probably not a major culture wars issue, even by sort of this time next year. But I think it’ll, you know, it will go from being this, I mean, it’s no longer even that niche, but still relatively niche thing worked on by a few think tanks to something that is increasingly something the general public are thinking about.
Dan Williams: Great stuff, great stuff. I think many of my predictions, to be honest, overlap with yours and probably a unifying theme is I expect all of these things to happen kind of gradually. So I think the systems will get better and better, not just in terms of how they perform on benchmarks, but in terms of their capabilities. But I don’t think we’re going to be seeing like a big bang upgrade this year.
On the stock market, I mean, there I think there might be a financial bubble that bursts, even though I take your point that AI itself as a technology is not going away. And I think people often conflate those two things in an important sense that they’re kind of orthogonal in the sense that it could be the case, and I think it definitely will be the case, that AI becomes increasingly impressive, capable, and integrated into the economy and into society more broadly. It could also be the case that given the financials of many of these companies and investment decisions, et cetera, et cetera, that you see some quite significant bursting of the bubble that has short-term significant economic impact. I’m probably kind of 50-50 on that, and I just don’t feel like I’ve got the expertise to really evaluate it.
All of the other things, I think you’re sort of directionally correct as they say. I do think one thing where again, I’m probably 50-50 is my understanding of all of the big AI sort of frontier labs at the moment is a major focus is on this kind of continual learning, building advanced AI systems of the sort that we’ve got today that can engage in continual kind of experience-based learning. So you don’t have to construct kind of bespoke reinforcement learning environments for specific tasks, but you can drop a system into an environment and it will be able to update its weights sort of continuously as it engages with that environment in much the same way that human beings and other animals can do.
Given that, at least to me as an observer, it seems like there’s so much focus on that and a recognition that that will be a really big change. I suspect that this time next year, maybe I’m sort of 50-50 here, that we will have seen at least one AI lab that’s made some significant progress on that. I don’t think it will be kind of immediate now they can do it, but I think maybe there’ll be a paper that’s released. Maybe there’ll be a kind of updated model that can do some version of this. And that would be a really huge, I think, story in terms of the historical development of these technologies. Other than that, I think I basically just agree with you. Directionally—
Henry Shevlin: Yeah, directionally correct is the best kind of correct. I know, yeah, I think there’s only not too much disagreement there between us. I mean, just to throw in one final thought, I wouldn’t be surprised, despite everything we’ve said, if AI is not the biggest story of this year. I think we live in an exceptionally unstable time, probably the most unstable time of my entire lifetime. And I wouldn’t surprise me at all if geopolitics or, I mean, particularly geopolitics, but potentially other domains create bigger surprises that swamp the relevance of AI, whether that’s war in the South China Sea, a major break between Europe and the US.
And that, you know, we’re focused here on AI, but I think that will have very big implications potentially for AI just because AI supply chains are so delicate. You know, a war in the South China Sea, for example, I think could be one of the biggest spoilers for most people’s AI timelines. So despite all my excitement around AI, I think given the sheer instability in the world right now, it may not end up being the biggest story of 2026.
Dan Williams: We are cursed to live in interesting times. Okay, that was such a fun conversation. We’ll see everyone in a couple of weeks.
