I used this prompt: “You are a world-leading academic. Please write an insightful review of this recent book published by Oxford University Press. The review should be suitable for publication in a top academic journal. It should be no longer than 2,000 words.”

I used the ‘pro’ setting for the amount of “thinking effort” (i.e., how much inference compute) the model uses, and turned off memory so it didn’t know it was talking to me, and so the book isn’t used as training data.

A few quick thoughts:

• The review is very good.

• If a human wrote it, I would be impressed not just by their detailed reading of the book but also by their intelligence, depth of knowledge, and insight.

• There are some annoying AI-isms in the writing style (“not X but Y”).

• The review’s objections are reasonable, although I think I could address them. (I also tend to have this feeling whenever humans criticise my work…)

• We are now at the point where people are simply kidding themselves if they think frontier AI models lack genuine intelligence, understanding, insight, or academic utility. I’m unsure exactly how research should change now that we have access to smart expert intelligences (that will only become more impressive and useful with time), but I am certain it must.

Subscribe now

Delusion and Epistemic Membership

Daniel Williams, Sam Wilkinson and Kengo Miyazono, The Social Roots of Delusions. Oxford: Oxford University Press, 2026.

The deepest provocation of The Social Roots of Delusions is not that social life can make us irrational. It is that remaining in contact with reality is itself a precarious social achievement. Williams, Wilkinson and Miyazono ask readers to reconsider the contrast between the isolated person whose beliefs resist correction and the community whose members collectively sustain an extraordinary conviction. These are not manifestations of a single underlying pathology. Rather, they reveal different vulnerabilities of epistemic interdependence: the loss of access to support that normally constrains individual error, and the mobilisation of that support to protect collective error. The book’s considerable achievement is to make these possibilities illuminate one another without requiring a uniform explanation of delusion.

Its three parts address classification, popular delusions and clinical delusions. The first advances the most philosophically revisionary claim. Drawing on Edward Craig’s function-first approach to knowledge, the authors ask what attributing delusion does within social life. Their answer is that it expresses “folk-epistemic bafflement” and identifies someone as a “rational lost cause” concerning a particular belief. Attribution does not merely identify an unreliable informant; it recommends abandoning ordinary attempts at rational correction. This genealogy loosens the conceptual connection between delusion and pathology and accommodates substantial heterogeneity among the beliefs attracting the designation. Delusionality becomes a social-epistemic status rather than an intrinsic property awaiting discovery (pp. 54–70).

Part 2 explains how otherwise capable individuals sustain “popular delusions”. The target is not religion, ideology or conspiracy belief indiscriminately, but a cluster comprising pronounced yet selective epistemic irrationality, compartmentalisation, emotional attachment and distinctive sustaining practices. The proposed explanation centres on socially motivated cognition. People can internalise claims they are motivated to propagate, while convictions can signal allegiance and cooperativeness. Sincerity is therefore compatible with strategic social function; the account is not a theory of universal cynicism. Crucially, beliefs need not be comforting to be advantageous. A terrifying narrative can justify hostility, secure standing or demonstrate loyalty even while making its adherent miserable. The distinction between wanting a proposition to be true and wanting the benefits of believing it is illuminating.

The account becomes most compelling in Chapter 5. Motivated cognition faces a rationalisation constraint: people cannot simply decide to believe whatever serves their interests. They need apparent grounds. The authors show how communities can collectively satisfy this constraint through “epistemic reward systems”. Dissent is discouraged, favourable evidence amplified and intellectual labour rewarded for producing increasingly persuasive defences of preferred conclusions. These practices also alter higher-order evidence: the apparent conviction of trusted others becomes further reason to believe. The discussions of white supremacy and QAnon illustrate how radically different formations can sustain such processes (pp. 142–148). Collective irrationality here is not merely individual irrationality multiplied. It involves the social production of an evidential environment within which otherwise implausible beliefs acquire credibility. This moves the analysis beyond familiar appeals to gullibility or an absence of critical thinking: considerable critical ingenuity may be invested in keeping a conviction intact.

Part 3 resists an equally familiar reduction in the opposite direction. Acknowledging delusions’ social character does not require locating their cause in a defective social-cognitive module. Against the “suspicion system” approach discussed in Chapter 7, the authors argue that detecting concealed intentions demands access to too much contextual information to be plausibly assigned to a highly encapsulated mechanism. Their alternative builds on multifactorial approaches, especially Daniel Freeman’s work, to explain three phenomena: the importance of social adversity, the recurrent social themes of delusions and resistance to social evidence. Perceptual disturbance, cognition, affect and behaviour remain indispensable; sociality specifies the circumstances in which these factors acquire their explanatory significance.

The treatment of persecution is particularly effective. Suspicion can generate anxiety and withdrawal; withdrawal can diminish corrective contact and increase vulnerability; these consequences can intensify the original suspicion. Protective responses thereby become constituents of a damaging feedback process (pp. 212–215). The distinction between “misfunction” and “malfunction” is important here: a mechanism can operate unsuccessfully because its inputs or conditions are abnormal, without being intrinsically damaged (pp. 247–248). This is not an argument against neurobiological explanation. It is an argument against assuming that every explanatory contribution must identify a further defect inside the person.

Chapter 9 sharpens the account by distinguishing testimonial isolation from testimonial discount. Evidence may fail to reach someone, or it may arrive but be assigned little weight. The distinction prevents apparent resistance to correction from being treated automatically as an individual incapacity. Moreover, discounting testimony can reflect ordinary epistemic vigilance under extraordinary conditions: apparently compelling personal experience may outweigh statements from people whose competence or sincerity is distrusted. The authors’ attention to these asymmetries makes “insensitivity to evidence” look too coarse a description. The relevant questions concern which evidence, delivered by whom, under what conditions of trust.

The resulting synthesis is substantial. Its originality lies less in announcing that delusions are social than in connecting social motives, distributed evidence and classificatory practices. Nevertheless, explanatory inclusiveness has costs. The authors candidly acknowledge that their framework remains largely qualitative, that mechanisms of compartmentalisation require clarification and that several central hypotheses await direct testing (pp. 275–277). The important next step is not merely to add further contributing factors, but to specify contrasts that discriminate among explanations. Under otherwise comparable conditions, when should changing the source of testimony alter conviction more than changing its content? Which features of a community’s reward structure make it responsive to correction rather than simply cohesive? Such questions would turn the framework’s breadth into testable predictions.

There are also conceptual difficulties that empirical elaboration alone will not resolve. First, the genealogy of attribution does not establish the authors’ stronger non-descriptivism. They maintain that delusion attributions are appropriate or inappropriate rather than accurate or inaccurate (p. 65). Yet an utterance can regulate social conduct while also describing a condition. A warning can direct attention and make a truth-apt claim about danger; similarly, an attribution might recommend a response while making defeasible claims about someone’s evidential responsiveness. Nor does the relational character of a status prevent its objective description. The absence of a single natural kind likewise need not eliminate descriptive constraints. The argument establishes that delusion attribution is not merely descriptive more securely than it establishes that it is non-descriptive. A hybrid account remains available.

Bafflement raises a related issue. The authors distinguish sincere bafflement from appropriate bafflement, recognising that inadequate empathy can lead us to overestimate incorrigibility (pp. 66–67). This qualification is essential, but it returns explanatory weight to judgements about the person and the interaction. Consider a clinician who recognises a familiar delusion without being baffled and responds by seeking a different basis for engagement. Such a case need not defeat a genealogy of ordinary usage. It does, however, suggest that identifying one important social function cannot exhaust the concept’s clinical and epistemic possibilities. The authors’ acknowledgment that not all uses fit their proposal makes this boundary especially consequential.

Second, the analysis of popular delusions leaves the distribution of irrationality insufficiently settled. Part 2 explicitly targets genuinely irrational belief systems, not merely beliefs outsiders dislike. Yet Chapter 5’s success in explaining how communities manufacture apparent warrant strengthens the possibility that some participants respond rationally to misleading evidence. The authors recognise this, describing insiders’ beliefs as less irrational than they appear externally (pp. 149–150). But the concession potentially goes further. Imagine a newcomer who receives convergent testimony from apparently credible sources, lacks access to the processes selecting that testimony and has no adequate reason to suspect manipulation. The community’s belief-forming arrangements may be epistemically corrupt without this participant’s acceptance being irrational.

An adequate account must therefore distinguish those who produce rationalisations, those who enforce conformity and those who inherit the resulting informational environment. These roles can overlap, but their epistemic defects need not. It must also distinguish a belief’s motivated acquisition from its subsequent maintenance on apparently good grounds. None of this vindicates popular delusions collectively. It does mean that identifying socially motivated processes somewhere in a network does not establish the irrationality of every believer downstream. The authors could restrict their target to participants who independently exhibit the stipulated epistemic defects. But identifying those defects would then be a separate empirical task, not something established by the social explanation itself. The book’s own social epistemology calls for a more differentiated reconciliation with the rational-social-learning explanations it contests.

Third, reading the account of attribution alongside the account of maintenance exposes a normative problem. The authors themselves identify the crucial feedback: labelling someone delusional can discourage testimonial interaction, deepen isolation and thereby help sustain the beliefs that occasioned the label (p. 243). They carefully qualify this claim; attribution is neither a sufficient nor an unmediated cause. Nevertheless, a practice whose function is to recommend disengagement may partly produce the conditions that make disengagement seem warranted. The apparent incorrigibility of a person’s belief cannot then be evaluated independently of how others have approached that person. This is not an objection to describing an exclusionary practice. It is a demand that the account’s own norms of appropriate attribution incorporate the practice’s causal effects.

The distinction required is between withholding assent and withdrawing epistemic recognition. Rejecting a proposition need not mean treating its speaker as incapable of contributing to understanding the situation. Likewise, declining repetitive contradiction need not amount to abandoning rational engagement. The book’s perspectival question—unpersuadable by whom and according to which norms?—should therefore extend to the quality, duration and institutional conditions of the relationship. Its postponement of sustained ethical analysis to future work (pp. 277–278) leaves unfinished something internal to the theory: how to distinguish warranted limits on argument from an exclusion that intensifies the problem.

The discussion of folie à deux supplies a complementary qualification. Here intense trust within a relationship can coexist with isolation from, or discounting of, outsiders (pp. 254–258). Thus neither increased connection nor increased trust is intrinsically corrective. What matters is their organisation: who can be heard, how credibility is allocated and whether a relationship permits revision. Read together, the book’s accounts of popular and clinical delusions suggest that epistemic health cannot be measured simply by conformity to a community. Communities themselves must remain answerable to evidence and challenge.

These reservations qualify rather than diminish the book’s importance. Williams, Wilkinson and Miyazono offer a powerful framework for relating processes too often considered separately, while making clear why neither pathology nor irrationality alone settles the explanatory task. Their strongest contribution is not a final account of what delusions are, but a demanding account of what explaining them requires. The unresolved tension between epistemic correction and epistemic exclusion is itself productive. A theory that understands knowledge as socially sustained must explain not only how communities protect themselves from error, but how they can do so without abandoning the people whose beliefs they find hardest to understand.

Epistemic status: big picture, speculative; the long-form writing equivalent of thinking out loud.

Recently, I told a friend I had been reading a philosopher’s writings about AI. He scoffed: “What does a philosopher know about AI?”

I could understand where he was coming from. His reaction illustrates a popular line of reasoning that might be put like this:

• AI is a complex technology.

• To think seriously about AI, one must therefore have deep technical expertise in how this technology works.

• Those who have this expertise are the “AI experts”.

• Those who lack it but write or talk about AI anyway are either amateurs or bullshitters.

This aligns with how many people think about knowledge and expertise in the modern world: Reality decomposes into narrow domains. So, to learn about the world, we should listen to those with certified expertise in those domains.

Most people who think this way allow that other forms of expertise are sometimes necessary. In thinking about AI’s impact on the labour market, for example, we should probably listen to economists. But even here, there is often an assumption that technical AI experts are uniquely well placed to pontificate.

I think that this whole perspective is confused.

Although deep technical expertise in how AI works is indispensable for answering many questions about AI, it is rarely sufficient and often unnecessary. Most questions about AI simply aren’t about AI in a narrow, technical sense. These include not just questions about how AI will transform our societies, but even questions more directly focused on AI itself, a uniquely mysterious technology that invites deep questions about the nature of intelligence, motivation, and agency.

When technical AI experts speculate about these issues, they often draw on extraneous and mistaken beliefs about psychology, sociology, economics, culture, political institutions, philosophy, and much else.

This suggests that AI discourse would improve by engaging a wider range of expertise, but I think that’s only half-right. The AI revolution threatens the existing division of intellectual labour and specialisation that we rely on to make sense of the world. Not only does it throw up novel challenges for which we simply lack mature disciplines, but it may transform so many parts of the world all at once that it undermines a core assumption of modern intellectual inquiry: that reality decomposes into relatively modular domains that experts can investigate independently.

At least, these are the claims that I will argue for here.

I will also extract an important lesson. Framed optimistically: the AI revolution is a golden age for polymaths and philosophers—thinkers who can escape the straitjackets of existing disciplines and engage in creative intellectual synthesis and big-picture speculation. Framed pessimistically: we must come to terms with moderate epistemic despair. As this new technological revolution unfolds, our world and future will become less knowable and navigable.

To Think About AI, You Must Think About Not-AI

Consider, first, the Industrial Revolution.

In one sense, this was a technological revolution that enabled humanity to unleash unprecedented amounts of mechanical power. Nevertheless, as Will MacAskill and Fin Moorhouse point out, most questions it raised weren’t narrow questions about the workings of steam engines or spinning jennies.

The Industrial Revolution transformed everything, including political and legal institutions, the structure of the economy, class and status hierarchies, human psychology, values, culture, gender relations, family structure, personal identity, ideologies, religion, and art, all of which interacted with each other in highly complex ways.

It was a technological revolution, but not only a technological revolution. The great thinkers who grappled with these changes—Ricardo, Mill, Marx, Durkheim, Weber, and so on—didn’t distinguish themselves with fine-grained technical knowledge about the mechanics of steam engines.

The AI revolution will be similar, except that the transformations will likely happen in a much more compressed timeframe.

An Interlude: Transformative AI Is Coming

Many people today are sceptical of this forecast, dismissing it as the hype and bullshit of self-serving tech bros and weird subcultures of “rationalists” and effective altruists. I won’t pause here to explain why this assessment is so misguided (others have already done so at length), except to briefly say this:

The brain is a complex information-processing mechanism. So, it should be possible to build computing machines that don’t just match but exceed human cognitive capabilities. Putting it this way undersells the transformative potential of AI, however, because it pictures AIs that are simply more impressive versions of us. In reality, the space of possible intelligences is vast, and AI systems may differ from humans in many profound ways, including the ability to be copied and run in parallel, operate at superhuman speeds, and modify their own designs.

These points were already clear to pioneers of computer science and AI in the 1950s, when one could reasonably have been sceptical. But today, we have built AIs that outcompete humans across a wide and expanding range of cognitive tasks, and there are huge economic, military, and scientific incentives to continue this progress. We should therefore expect to build AIs in the coming years far more powerful and capable than those we already have, including not just digital systems but advanced robots. And eventually, we will build machines more capable across all domains than the most capable humans.

These points hold even if one thinks that the abilities of advanced AI systems today are highly “jagged”, that the current AI paradigm will hit a wall, that we are in a financial bubble, that concepts like “AGI” (artificial general intelligence) make little sense, or that AI’s impacts will be slowed by human, organisational, and institutional bottlenecks and speed limits.

In other words, although one can reasonably debate how close we are to transformative AI, what forms it will take, and how quickly it will diffuse through our economies and societies, we should be highly confident that it is coming, and take very seriously the possibility that it will arrive within our lifetimes.

Even Many Questions Directly About AI Aren’t About AI

The Industrial Revolution also highlights another, subtler point: the core technologies themselves often outran humanity’s scientific understanding of why and how they worked. The attempt to understand steam engines, for example, played a major role in the development of thermodynamics, which emerged long after steam engines had begun transforming the world.

A similar but more extreme situation arises with AI.

On one level, this is just the familiar point that even technical AI researchers lack a precise, mechanistic understanding of why and how deep neural networks work. However, this observation greatly understates how much more mysterious AI is than any technology that has come before. Even if we had satisfying mechanistic explanations of how ChatGPT or Claude work, how we should think about AI would still depend on a vast range of questions about the nature of minds, intelligence, and agency that would remain largely unsettled by this knowledge. Moreover, these questions are highly relevant for understanding how the AI revolution is likely to unfold.

Consider just some of the controversies that dominate discussions in this area:

• Will “super-intelligent” AI systems kill or disempower humanity?

• How easy is it to “align” AI systems with human values?

• What capabilities must AIs have to match human cognitive performance across all domains?

• What can superhuman reasoning power enable a system to achieve in the real world?

• Does it make sense to talk of intelligence as a single scalar property that systems have more or less of?

• Is the concept of “Artificial General Intelligence” useful or even meaningful?

• How much room above the peak of human performance exists for different capabilities, such as persuasion and forecasting?

• How should we measure the rate of AI “progress”?

• Could current or future AI systems have conscious experiences?

These are questions directly about AI, much more so than questions about how AI will transform our relationships, economies, and cultures. But although technical AI expertise is relevant to many of them, they go far beyond technical matters.

Thinking About (Thinking About) AI Doom

Consider just the topic of AI takeover, the popular threat model according to which advanced AI systems will eliminate or disempower humanity. Although some discussion of this risk is informed by technical details about how modern AI systems work, much of it is driven by high-level theoretical assumptions, analogies, and thought experiments.

For example, the most influential argument in this area draws on the concept of “instrumental convergence”, an alleged tendency of intelligent systems to converge on instrumental goals like self-preservation and power-seeking for a wide range of possible ultimate objectives. This is taken to demonstrate that systems tasked with seemingly benign objectives like creating paperclips might end up pursuing instrumental goals that conflict with our interests and even survival (e.g., using our bodies as raw materials for paperclip factories).

Two other influential arguments allege, first, that “aligned” or “friendly” super-intelligent systems comprise a tiny fraction of possible super-intelligent systems, implying that misaligned AIs are much easier to build than aligned ones, and second, that there are in-principle difficulties in training machines to be aligned instead of merely appearing aligned until they accrue more power.

These arguments attempt to derive conclusions from first principles about the nature of agency, intelligence, and learning, and they were advanced before modern large language models were invented.

Of course, one might reasonably respond that things have changed today: we now have many technical AI safety researchers and an extensive empirical literature that bears on these worries, including clear demonstrations of “alignment faking”, deception, and other worrying behaviours in AI models.

Nevertheless, the highly theoretical arguments advanced by figures such as Eliezer Yudkowsky and Nick Bostrom continue to play a huge role in shaping how even AI safety researchers interpret specific findings and frame threats. For example, whether researchers treat an AI hacking into a private company as a rare, fixable outcome of current training methods or as a harbinger of future doom depends largely on their pre-existing beliefs. That is, the interpretation of the findings is as much informed by pre-existing theory as the theory is informed by the findings. This explains why researchers exposed to the same findings can disagree so much about the likelihood of future AI takeover, in contrast to fields such as climate science, where high levels of expert consensus reflect the much greater role of rich bodies of empirical data and decades of confirmed empirical predictions.

If you’re a hard-nosed empiricist who thinks knowledge should be rooted in empirical findings and peer-reviewed scientific consensus, and who scoffs at elaborate first-principles reasoning and speculation about complex systems, you will probably not be very impressed by these arguments.

In contrast, the self-styled “rationalists” most worried about AI doom, such as Yudkowsky, have an extremely optimistic view of the powers of pure reasoning. Hence, they are happy to reach high levels of confidence in bold forecasts even when direct data is sparse, based on complex chains of abstract theoretical reasoning alone.

Whatever one thinks about these disagreements, they further illustrate my point. One of the central questions about AI connects not only to high-level theoretical issues about intelligence and agency but to epistemological debates about the powers of pure reasoning.

In fact, one of the most fascinating aspects of the AI takeover debate is how these epistemological disagreements feed back into how people think about the power of AI systems themselves. The rationalists most worried about AI takeover are so worried, at least in part, because they assume that a machine with superhuman intelligence and reasoning abilities would have immense power, including the ability to take over the world and wipe out all of humanity.

In contrast, those who are sceptical of the power of speculative rational argumentation to deliver substantial knowledge about the world are also typically sceptical that merely endowing machines with more cleverness and reasoning ability will automatically endow them with real-world power.

In other words, our beliefs about what we humans can accomplish through pure reasoning feed into our expectations about how powerful pure reasoning will be in machines.

On Coping With Technological Revolutions

An obvious corollary of all this is that technical expertise does not, itself, confer expertise, or even special insight, on many of the huge challenges raised by AI. We can’t simply defer to “AI experts”.

The classic illustration of this point is Geoffrey Hinton’s 2016 recommendation to stop training radiologists on the grounds that AI systems would soon outperform them at image classification. Hinton, a “godfather” of AI, was highly prescient about trends in AI capabilities. But he coupled this expertise with a low-resolution folk model of work and labour economics. Economists will tell you that, like almost all professions, radiology involves a complex, integrated bundle of tasks and responsibilities, and that radiologists operate in a real world in which technological diffusion is slowed by regulations, liability regimes, and numerous other frictions and speed limits.

Ten years later, not only do radiologists still exist as a profession, but demand for their work is apparently rising.

Many philosophers and neuroscientists would make similar complaints about Hinton’s arguments concerning AI consciousness, and many experts on persuasion and belief formation cringe at some of Hinton’s forecasts about AIs’ superhuman persuasion abilities.

One simple lesson you might draw from these reflections is that to grapple with the AI revolution, we need to engage with a wide range of academic disciplines. Computer scientists won’t suffice; we also need economists, psychologists, sociologists, political scientists, philosophers, and so on.

In some sense, this is obviously right.

However, this response can also fail to grapple with the severity of the epistemic challenges ahead. Appeals to the importance of multiple forms of expertise often assume that problems can be divided into existing disciplinary buckets, that relevant knowledge already exists within these disciplines, and that we can address these problems by simply integrating such knowledge.

These assumptions are likely too optimistic.

A New World Requires New Knowledge

Consider questions that seem absurdly science-fictional to many now, but will soon acquire monumental importance: How should we think about institution design, ethics, and social justice in societies that contain vast numbers of digital minds and robotic agents? How should we understand the collective dynamics of agentic AI “swarms”? What happens to science, research, and technological development when AI systems can substitute for much of human cognitive labour?

For these and countless other questions, many of which we haven’t yet even anticipated, bodies of knowledge, concepts, and methods across existing fields of research will obviously be important. But it should be equally obvious that these questions throw up fundamentally novel kinds of puzzles and challenges that existing disciplines aren’t well-equipped to handle. The knowledge we have accumulated about individual and collective behaviour has targeted human agents (or, at most, organisms). Even when we have constructed more idealised models of rational agency, coordination, competition, conflict, and institutions, they have been rooted in our experiences of human agents.

Advanced AIs won’t just have a wide range of unprecedented capabilities in things like copying, parallelism, processing speed, high-bandwidth communication, and self-modification. They will also have new kinds of goals and motivations (if those terms are even applicable) shaped not by ruthless Darwinian evolution but by complex mixtures of intentional design, curated training data, human feedback, and economic and geopolitical pressures.

At present, we simply lack mature sciences of what might happen when millions or more artificial intelligences with superhuman capabilities and deeply inhuman goals enter our societies, cultures, economies, and institutions. To make progress on the many explanatory and normative questions this will raise, we will need new concepts, models, theories, and fields.

The Prospects of Decomposition and Specialisation

A deeper issue concerns the prospects for intellectual specialisation under conditions of radical technological transformation.

As with wealth creation, the advancement of human knowledge is closely connected to a complex division of labour. If we had to create all our knowledge individually, we would know almost nothing. So, we specialise in acquiring narrow forms of expertise and then share the fruits of our intellectual labour with others, enabling cumulative advances in the frontier of knowledge and understanding over time.

Modern science and academic research formalise this division of labour. To simplify greatly: economists study the economy, political scientists study political institutions, sociologists study social relations, and so on.

Of course, people often complain about this modern balkanisation and siloing of research. Don’t most big questions straddle multiple disciplines? Why can’t we go back to the good old days before such extreme intellectual specialisation, when thinkers like Smith or Weber or Durkheim engaged with huge questions about humanity, society, and culture?

In some ways, these concerns are legitimate, but they are also naïve. If we want to understand the world, there is simply no alternative to narrow specialisation and the division of intellectual labour. Figures like Smith or Marx could pontificate about huge, general questions because they were writing when nobody really knew anything. Today, we have built mature research fields across many narrow domains, and to the extent that interdisciplinary research is even possible, it is by integrating the fruits of this specialisation, not by replacing it.

An important question, though, is why this specialisation works. After all, society really does seem to be a vast, unimaginably complex, integrated whole. How can people make progress in studying any one part without also engaging with the broader system?

In a classic article on the “architecture of complexity”, Herbert Simon provided an important answer: complex systems become comprehensible when they involve a kind of near-decomposability—subsystems with strong internal interactions but weak interactions with each other, at least over short timescales. These weak interactions mean that other aspects of the broader system can be, to some extent, bracketed off as stable background conditions. That is, although highly specialised inquiry doesn’t require independent domains, it does require interactions between those domains to be sufficiently weak or slow that researchers can treat the rest of the system as relatively constant or “exogenous”.

The AI revolution may threaten this condition. As advanced AI systems and robotic agents are rolled out, they may transform many parts of the social system all at once, eroding the decomposability on which disciplinary specialisation depends. Economic changes may transform political power and public opinion; these changes may reshape ownership and regulation; changes in the information environment may affect democratic preferences; and all of these changes might feed back on one another and countless other forces, including the trajectory of AI itself. Such disruptions needn’t be a permanent condition of our future, but deep technological revolutions can create a transitional period in which so many aspects of society change so profoundly, and interact so strongly, that background conditions specialists hold fixed no longer remain stable.

An Illustration: The Economics of Transformative AI

Consider, for example, one of the most important questions about transformative AI: how will this technology affect the economy? Should we expect mass unemployment? Will it unleash unprecedented amounts of economic growth? How will the concentration of wealth be affected?

Famously, economists and AI specialists are often exasperated with each other on these questions. Economists complain that technologists routinely ignore elementary concepts within economics, such as comparative advantage, bottlenecks, and cost disease, whilst technologists respond that economists treat AI as just another technology, ignoring the very features that make it so transformative, including its potential to one day substitute for all human labour.

One response to this kind of disagreement is to plug truly transformative AI into standard economic models and explore what happens. A fascinating and important literature now does just this, treating AI as a kind of reproducible capital that can replace human workers across all tasks. As you might expect, the models often output insane conclusions: once growth rates are no longer bottlenecked by the population of human workers, they can explode, alongside huge increases in inequality and the near-disappearance of labour’s share of income.

We absolutely need more of this kind of work, and the insanity of its outputs is a point in its favour. We should expect a crazy future.

And yet, the models picture worlds in which transformative AI can simply be inserted whilst a whole range of background conditions—property rights, ownership structures, firms, and so on—remain fixed.

This seems very implausible. One needn’t be a Marxist to appreciate that profound technological change (changes in the “forces of production”) is very unlikely to leave the core political-social-ideological structure intact.

Transformative AI won’t just be an input into our existing system of bureaucratic capitalism any more than industrial technologies were merely an input into the economic structure of the agrarian and feudal economies that preceded it.

In periods of relative stability, we can treat “the economy” as a stand-alone object of study, bracketing off the political institutions, laws, values, incentives, and public opinion that underpin its existence. In deep and wide-ranging technological revolutions, this stability breaks down. In this sense, just as most questions about AI are not only about AI, so most questions about AI and economics are not only about AI and economics.

A Golden Age For…

There are many lessons one might take from these reflections. I will briefly mention just three, although they are really three framings of the same lesson.

… Polymaths

First, the AI revolution will be a golden age for polymaths. If the most important questions it raises cut across existing disciplines and often reach far beyond them, it should increase the value of theorising and research that escapes the constraints of narrow academic specialisation, synthesising ideas and findings from different domains, and proposing fundamentally new concepts, frameworks, and even academic fields.

Admittedly, this kind of activity carries obvious risks. In some sense, true polymathy is simply impossible today. Researchers can’t even fully master the body of knowledge in their own fields, let alone multiple ones. And in practice, lots of “bold” intellectual syntheses and speculations are amateurish and pointless.

Still, if the world transforms in ways that don’t respect existing academic specialisations, we must confront the epistemic challenges this creates.

… Philosophers

It will also be a golden age for philosophers.

In some sense, it’s obvious that philosophy is relevant to many of the questions raised by AI, from questions about machine consciousness and moral status to debates about what a just society looks like in a world with transformative AI. Historically, philosophy and AI have always been closely entangled. Alan Turing’s classic 1950 “Computing Machinery and Intelligence” was published in Mind, one of philosophy’s flagship journals. It begins with the question, “Can machines think?”

Philosophy’s relevance to the AI revolution is even broader than this, however. As a discipline, it is fundamentally concerned with important questions that aren’t yet owned by mature scientific disciplines, typically because they touch on deep conceptual and theoretical puzzles that we haven’t solved or involve ethical or political issues that science alone cannot settle.

Consider two questions: “How does the visual cortex in the brain work?” and “Are shrimp conscious?” The first is a question for neuroscience; the second is more recognisably philosophical. This isn’t because neuroscience and other forms of scientific research aren’t relevant to the latter question. They’re highly relevant. But we don’t—yet—know how to turn the question into a tractable one that might be answered by ordinary scientific methods. It is connected to difficult conceptual, theoretical, and metaphysical questions that humans remain deeply confused about.

People sometimes ask why, unlike science, philosophy doesn’t make progress. This brief reflection illustrates why the question often rests on a semantic mistake. Once a discipline starts to make straightforward scientific progress, we simply stop calling it “philosophy”. Philosophy is our oldest discipline. Pretty much all questions about reality and our place in it were once philosophical questions. Over time, we have turned many of these questions into tractable scientific puzzles and research programmes, giving rise to fields like physics, biology, and psychology. Philosophy is largely the questions that remain.

AI will raise the value of philosophy in this sense. At present, no observations, experiments, statistical analyses, or scientific models could straightforwardly answer how we should live alongside artificial minds, whether advanced AI will disempower humanity, or what the state and economy will or should look like when we have AIs that can outcompete humans at everything.

This doesn’t necessarily mean a golden age for professional philosophy, which can be as myopic as any other area of modern research. To be useful, philosophy must be non-dogmatic, informed by science, and engaged with real-world challenges. Much professional philosophy today doesn’t meet those standards, and much of the best philosophy that does isn’t done by professional philosophers.

Philosophy is an activity, not a guild.

… And Epistemic Despair

A golden age for polymaths and philosophers is not a golden age for knowledge and understanding. If anything, the reverse is true: bold intellectual syntheses, creativity, and philosophising rise in value precisely as the world becomes more chaotic and confusing.

A division of intellectual labour into specialised fields is a major reason modern science is so effective. It makes learning about the world a relatively simple matter of trusting what specialists in specific domains agree on.

Admittedly, this epistemic order performs worse than many would like to acknowledge. The feeling of understanding often outruns its reality. We are much better post-hoc storytellers than forecasters. But still, if we step back and reflect on how much we can know today, it is a remarkable success story.

The AI revolution threatens it. Not only are the certified experts not experts on most of the questions that matter, but it’s often unclear who even would be. We lack mature sciences for many of the unprecedented challenges and opportunities we will confront. The technology will affect so many different aspects of society at the same time that society’s current decomposition into modular parts may break down, at least before a new decomposition—a new regime—emerges from the ashes.

Narrow knowledge will still be attainable: how specific AI systems perform on benchmarks, how workers use them, whether they are more persuasive than human debaters, and so on. But as we move from these local, technical issues to more general, high-level questions about the coming transformations, our confidence should decrease sharply.

***

Writing about AI’s impact on the economy, Derek Thompson says, “I can’t emphasize enough that ‘nobody knows anything’ is about as close to the reality here as three words are going to get you”:

Nobody [knows] what’s going to happen this year, or next year, or the year after that. There is no secret cigar-filled room of people who have unique access to some authentic postcard from the future. When you drill down underneath the bluster, the boosterism, the fear, the anxiety, what’s there at the bottom is genuine uncertainty, a vacuum into which storytelling is flooding. The frontier labs don’t really know what they’re building exactly, and economists don’t really know how to model the thing they claim they’re building (genuine recursively self-improving AI agency isn’t really analogous to something we know about).

“Nobody knows anything” isn’t quite true. Beyond the narrow questions already mentioned, I think we can know at least one very big thing with a high degree of confidence: transformative AI is coming, and maybe soon, even if its shape, timelines, and trajectory are mired in radical uncertainty. Moreover, not all speculation and opining are equally valuable. We are all ignorant, but some are more ignorant than others.

Still, Thompson is clearly onto something. The disagreement, confusion, and storytelling that characterise AI debates today aren’t just a product of ordinary human biases and shoddy reasoning. The future itself is largely opaque. And if this feels true now, as we sit in the foothills of the singularity, try to picture the epistemic vertigo we will experience once the AI revolution really gets going.

Return to my friend’s question: “What does a philosopher know about AI?”

Not a lot, and certainly not enough. In that sense, philosophers are in good company.

In this episode, Henry and I speak with Fin Moorhouse, one of the most interesting thinkers on how we should understand - and prepare for - the transition to truly transformative artificial intelligence.

Last year, Fin and Will MacAskill published “Preparing for the Intelligence Explosion”, which explores what might happen once AI systems start to dramatically expand the amount of intelligence in the world. In my view, this is one of the most important essays to master if you want to understand just how crazy the world might become in the near future due to rapid AI progress, and the huge challenges and opportunities this will create.

Fin currently works as a researcher at Google DeepMind, but he joins us in a personal capacity. The views he expresses are his own.

Among other things, we discuss:

• What an “intelligence explosion” actually means and how likely it is.

• Whether AI must substitute for human researchers, rather than merely augmenting them, to trigger explosive growth.

• Why current AI systems may lack research taste - and whether scientific progress depends on ambition, status-seeking, and irrational confidence in one’s own ideas that purely truth-seeking AIs would lack.

• The role of experiments, learning by doing, institutions, embodiment, incentives, and large-scale social coordination play in technological progress.

• Robotics, self-replicating capital, and the alarming possibility of an industrial explosion.

• Techno-optimism and techno-solutionism.

• Why solving alignment isn’t sufficient to ensure a flourishing and enlightened future with superintelligent AI.

• Why superintelligence might make questions of space governance extremely important.

• Fin’s deflationary “illusionist” view of consciousness and Henry’s objections to it.

• Whether we need to settle metaphysical controversies about consciousness before we address ethical and political questions about AI welfare, rights, and moral status.

I really enjoyed this conversation, which, in addition to introducing Fin’s perspective on a range of questions, includes a wide range of disagreements and debates as well!

Fin is one of the smartest and most thoughtful people working on questions that are shockingly under-researched given their monumental importance.

Subscribe now

Links and further reading

• Fin’s website and writing (and Substack)

• “Preparing for the Intelligence Explosion” - Fin Moorhouse and Will MacAskill’s.

• “No ghost in the machine” - Fin’s essay on consciousness, illusionism and digital minds.

• “Will We Really Put Data Centers in Space?” - Avi Parrack and Fin on the technical, economic, and political case for putting data centres in space.

• “The Adolescence of Technology” - Dario Amodei’s explanation of his “country of geniuses in a data centre” framing for understanding AI.

Transcript

• This transcript is lightly AI-edited and may contain minor mistakes.

Dan: Welcome back. Today we’re joined by Fin Moorhouse. Fin is one of the most insightful thinkers working on how we should understand — and also how we should navigate — the transition to a world with truly transformative artificial intelligence. Fin, last year you published an excellent article with Will MacAskill on preparing for the intelligence explosion. What’s the central argument of that essay?

Fin: Yeah, thanks for having me, both — fan of the pod. The argument is: first of all, something like an intelligence explosion — and I can say what that means — seems really quite plausible, potentially really quite soon. Secondly, that could just have very transformative effects, and effects in lots of different domains that matter — so not just isolated to one very kind of neat, easily analysable kind of transformation. And then lastly, that we should be trying to do stuff to prepare for this kind of wave of transformations that we might expect from AI. And we should be trying to do that now.

And as a corollary, there isn’t one magic bullet here, right? In the same way that when you think about the Industrial Revolution, it’s very hard to think of any particular thing that any particular person could have done to make things go better — but it still might have paid to do lots of things and think about lots of different upshots earlier rather than later.

Dan: So you mentioned there at the beginning this term intelligence explosion. Do you want to say a little bit more about how you’re understanding that term? Because it carries quite a lot of baggage, I think, in terms of how people understand it. And I think the way in which you unpack it in the essay is actually very interesting, and in some ways requires less baggage, or sort of fewer assumptions, than some uses of that term.

Fin: Yeah, so we didn’t mean anything precise by that term — somewhat lazily, but also somewhat deliberately. Roughly speaking, I think what we had in mind was just a radical, dramatic, fast influx of machine intelligence in the world, especially when compared to the kind of sum total of human intelligence and ingenuity. So the thought is that at some point — again, in a hand-wavy sense, I’m not trying to pin this down precisely right now — but at some point, the kind of sum total of just intelligence represented by AI and non-human thinking could just surpass all of us combined. Now, there are different ways that could happen, which we can talk about — different ways to really try to operationalise that — but that’s the rough idea.

Henry: So the intelligence explosion is specifically an explosion in non-human intelligence, right? Because I think you could say: look, the amount of intelligence in our civilisation, in the world, has been rapidly increasing as we augment our intelligence. Even, you know, the internet is an example of a massive intelligence augmentation. So, you know, I could imagine someone saying: look, the intelligence explosion has been going on for thousands of years. Like, how is this different?

Fin: Yep — that’s a totally fair point. And actually maybe it’s kind of not necessary to — I used the word non-human, and maybe I shouldn’t have — because maybe in some ways you can think about AIs as augmenting, or as tools for, people. I think the distinctive thing then is that the kind of influx will one way or another come from machines, rather than the number of people, for example, or people learning to be smarter on their own. And you’re right that in some sense, you know, our modern history is a kind of drawn-out intelligence explosion. What we’re trying to emphasise is the speed, which I think — again, very plausibly, but not necessarily — could be unprecedentedly fast: like, many, many times faster than it’s ever been before.

Dan: So the thought is, at the moment, in terms of increasing the amount of intelligence in the world — and your focus in the paper, at least initially, is the amount of intelligence which is going into research, R&D, the kind of research that underpins technological progress — at the moment, we’re being heavily bottlenecked by just the size of the human population, which is not expanding at a really, really rapid rate. So I think you give an estimate in the paper: something like the rate of growth of research effort rooted in human intelligence is, like, four or five per cent per year, or something like that.

And the thought is: well, we’ve now got this other source of intelligence, which comes from AI. And at the moment, AI in terms of its capabilities doesn’t really match everything that the human mind can do when it comes to research. But at a certain point, you’re going to get AI systems that at the very least will have kind of parity with human capabilities. So now you’ve got this other source of intelligence, and because we can and because we will scale up the amount of that kind of machine intelligence very, very quickly, you get this, like, rapid expansion in the amount of machine intelligence, in a very kind of short period of time relative to what we’re used to on the kind of human timescale — you get this just explosion of the kind of research that then powers lots of technological progress. Is that a fair summary of what you argue in the paper?

Fin: That is such a good summary that I’m gonna struggle to improve on it, but I’ll try almost paraphrasing it at least. I think part of the thought here is — I mean, it sounds a little silly to say out loud, but why should we actually care about AI progress? One reason is because there are, you know, direct effects from AI — people directly interacting with LLMs — that has, you know, social epistemic upshots and so on. But those are not the only effects. Actually, I think that at least as important are the downstream effects, and in particular the kind of technological change, the new kinds of ideas, that AIs could generate, right? In the sense that you don’t actually have to interact directly with the AIs that are generating new technologies, in the same way that you don’t have to meet the researchers who develop technologies that influence your life profoundly. So that seems like a place to start thinking about what AIs could be doing.

Henry: Can I just, before we get too into details — there is a worry I have here that you’re sort of using intelligence as a mass noun, as though it’s something that we can cleanly measure. And, you know, look, I realise in the human case we do have actually pretty clean operationalised measures of things like fluid g. You know, people argue about IQ, but it’s fairly predictive. But I think as soon as we start — particularly as soon as we start moving outside the human case, right? Like, if you give me a crow and you give me a dog and say which one has more intelligence inside their head? That seems like an ill-posed question.

Fin: Yep, totally agree. So I hopefully don’t want to imply that I think intelligence is easy to measure. I mean, you can think about the kind of intensive sense — how smart are the smartest AI systems? — I mean, notoriously, famously kind of ambiguous and rough to operationalise. Also the kind of extensive sense — like you said, you know, intelligence as a kind of lump noun. Yep — I absolutely only intend to mean this in a very hand-wavy and extremely vague sense. And I think it’s important and useful to try to kind of pin down different particular ways to operationalise. But I think so far I’m trying not to go in for any kind of particular conception.

But — so, you know, if that kind of sounds somewhat reasonable — one way to get a handle on this question, like Dan was alluding to, is just to ask what drives technological progress historically. And you can write, as many have done, you know, bookshelves about the details. But a very high-level account that I find very compelling, from a kind of economics perspective, is: look, if you need a one-factor explanation of technological progress, it is in terms of the amount of research effort at a time, right? Which you can proxy by just the number of researchers. Charles Jones, the economist, is kind of best known for pushing on this view in the kind of mid-90s. Moreover, it becomes harder to make technological progress as you make more of it, because you pick the low-hanging fruit first. And so you kind of need to parameterise, like, the rate at which ideas get harder to find. But also, historically, we’ve just piled on more and more researchers. And so roughly we’ve sustained about a kind of constant rate of tech progress, at least in terms of measured productivity.

The reason that’s kind of convenient and nice is not just because it’s borne out fairly well empirically, but because you can just make this move, like Dan said: what if we could just massively increase the kind of human-equivalent researcher workforce with machines? You know, maybe they’re working as tools, right — they’re saving researchers time, and therefore kind of giving them more time on the important stuff. Maybe they are in some ways replacing them, substituting for them. Again, I actually don’t think at this kind of high-level place it really matters to get into those details. But if in some way they could speed up research radically quickly, that could then speed up tech progress.

Why does that matter? I think so many of the really hairy social, political questions that we’ve faced as a civilisation over the last century or two have been originated by just finding ourselves with new technological possibilities and asking ourselves: how do we deal with this? How do we kind of structure society around it? So maybe the most salient example is nuclear weapons, right? Fifteen years ago, no one had even thought of the nuclear chain reaction; now we have a working nuclear bomb. How the hell do we deal with this? That’s a really rough question. If that kind of process happens 10x as fast, maybe even faster, that starts to feel like a really hairy thing to deal with.

Dan: So there’s this model here, which is: increasing the amount of sort of machine-based cognitive labour, in the way that we seem to be on track to increase it in the coming years and decades — there’s just a very sort of strong case for thinking that that is going to dramatically speed up the rate of technological progress. I think that in and of itself raises lots of interesting questions, like: is that really the correct model of what drives technological progress? But I think it is worth just, sort of, firstly spending a little bit more time on, like, this question of what do we mean when we say improving the capabilities of AI, and kind of scaling up the amount of AI-generated cognitive labour that we have — that that’s going to lead to, like, a massive expansion in the amount of research that we can do. And I think one thing you’re saying is you get that result even if you don’t make a whole set of, like, really controversial assumptions.

So, for example, you needn’t assume that intelligence is, like, a simple matter, where, like, one thing is just as intelligent as another, or more intelligent than another. You might even get it if AI systems are just kind of augmenting human capabilities, not fully substituting for them. I guess my question there is: it’s sort of difficult for me to see how you’re going to get the really rapid increase in the rate of — let’s say research progress, or in the expansion of the amount of R&D that we can do — if AI systems are merely augmenting and extending human capabilities. Like, I totally understand the argument that says once AI systems can just flexibly substitute for human beings — so that you can get AI systems in some sense creating their own kind of autonomous domain of research — then just the numbers, in terms of how many AI systems we can get and how we might improve their capabilities and so on, that could result in this huge increase in the amount of research that we’re doing. That I buy. But the idea that you’re gonna get that merely by their augmenting human capabilities — that seems a bit more controversial to me.

Fin: Yeah, I think I do somewhat agree with that. So the example that comes to mind is maths, since it’s been on the, you know, on the Twitter feeds recently. You know, at some point around, I guess, the mid-twentieth century, it became effectively free to calculate. And as far as I understand, a reasonable fraction of a mathematician’s job before that point was to do numerical calculations by hand. You might naively think: now it turns out that I can calculate something like 10,000 times faster than I ever could before — that’s going to accelerate mathematics in this kind of, you know, radical singularitarian way. And it didn’t, right? Basically not at all. Similarly, you think about, you know, just the spreadsheet as an innovation across knowledge work: it was useful, but it didn’t meaningfully accelerate basically any kind of field of knowledge work. And also symbolic algebra as well, which kind of came later in maths — didn’t have any kind of radical effects.

So if the affordances that we get from AI are always complementary with other human capabilities, and we don’t get more of them, I basically agree: we could get a big one-time speed-up, or a one-time kind of, you know, boost, when we kind of pick all the fruit that were hard to pick before, and then we’re just bottlenecked again. And this is just in some way the story of any, you know, new technology — this is what normal technologies do. One way that AI could be different is if it could be always complementary in some sense — so always augmenting — but the set of tasks that it’s able to augment is always expanding, right? So, like, in some sense human researchers always have some kind of absolute advantage at something, but the set of those things is just kind of halving constantly. That would be one way that things could go where I think you do get a sustained speed-up. But I also just largely agree that for the more kind of dramatic speed-ups, you need — at least in lots of domains — it needs to look more like the AIs are basically capable of flexibly substituting for human researchers.

Henry: Just quickly on this sort of broader topic, since you mentioned — or referred to — the amazing, very cool maths discoveries we’ve seen really in the last six months, last year. There’s this question Dwarkesh Patel asked Dario Amodei a few years ago — I’m sure you know what I’m gonna say. You know, he said — I’ve got it here — what do you make of the fact that LLMs (this was 2023) have basically the entire corpus of human knowledge memorised, and they haven’t been able to make a single new connection that has led to a discovery? Whereas if even a moderately intelligent person had this much stuff memorised, they would notice: this thing causes this symptom, this other thing also causes this symptom — there’s a medical cure right there. So, like, I’m curious what your take is on why we haven’t already seen some amazing scientific outputs from LLMs — only in the domain of maths so far.

Fin: Yeah, it’s a great question, and that’s totally true. So I don’t know the answer at all. One thing that goes through my mind is — yeah, it seems like even the smartest AIs, they seem to really lack kind of research taste, right? So they’re very, very good at kind of knowing relevant results, at making kind of pedantic, you know, points. But when you think about what makes a very successful kind of superstar researcher in their field so successful, often they have this hard-to-pin-down sense for what research agendas are likely to pan out, right? Some of this, I think, comes from — you know, in the language of AI — being trained on non-public data, right? So, you know, imagine you’re a PI: you supervise a bunch of students, they’ll try a bunch of stuff, some of the ideas won’t pan out, some will. The ideas that don’t pan out don’t get put on the internet, don’t get written up as, you know, here’s our report of a failure. But you still get to learn from them.

I expect there’s much more going on there which has to do with, you know, creativity — and, you know, I think the fact that it’s actually quite hard to pin down what exactly is a missing ingredient suggests that we haven’t operationalised it well enough to make the AIs good at it. It may be that there’s no way around collecting very expensive-to-collect data from human researchers who have research taste, in order to teach the AIs that taste, and that could take some time. So the short answer is: I don’t know, but I expect it has something to do with some components of research skill which are basically independent of, you know, kind of crystallised knowledge.

Henry: Interesting. Not to put you on the spot, Dan, but I’m curious — you know, I think we’ve talked around this topic a few times. I’m just curious if you have a take on this — like, why we haven’t seen—

Dan: Yeah. So I think part of it is just to do with the capabilities of the systems. But there is this other aspect to this — and this is, like, a little bit of a digression, but I think it is interesting. I first came across this view from Hugo Mercier; I don’t think he necessarily kind of originated it. But it’s the idea that advancing the frontier of knowledge, in the human case, relies on individual-level irrationality. Because if you imagine you’ve got a community of people, and you’ve got an individual who just wants to figure out what’s true, almost always the most rational thing for you to do is just to defer to majority opinion or something — because it’s much more likely that other people would have figured out the truth collectively than that your own ideas are correct. But if everyone does that, then you just never advance the frontier of knowledge.

And so the way in which we get around this is: human beings just care a lot more about, like, status and glory and esteem than we do about getting things right. So from, like, an individual perspective, you often find people just placing so much weight on their own ideas such that they’re willing to just pursue them to an irrational extent. And one consequence of that is you just get loads of people backing themselves when their ideas are just completely false and ridiculous — which I think we do in fact see all over the place, including in science. But very occasionally you get people who, because they back themselves and they really care about getting the glory of being the one that makes the discovery, they actually kind of advance the frontier of knowledge. And I think if you do look at the history of science, you find a lot of the people that make really significant discoveries are kind of egomaniacs, in a way that that basic model suggests. But anyway — if you think that kind of view is correct, then you might think optimising AI systems to be really good at truth-seeking might in fact not unlock the source of knowledge creation in the human case.

Fin: Yeah — I mean, it’s super interesting. I can tell you kind of what goes through my head. So this seems really plausible for domains like social sciences, humanities and politics, where ideas succeed, you know, partly on their merits, but partly on how they’re communicated and how kind of fit they are to, you know, succeed in certain ways. You could imagine that just an AI system alone can have, in some sense, a good idea in politics or philosophy, but it’s bottlenecked by, you know, just its ability to persuade or, you know, kind of form coalitions. That’s something that — I don’t know — you’d, like, have thought about a lot, Dan, and I haven’t. My sense is, in hard sciences — you know, thinking about drug discovery or, like, solid-state physics or whatever — it’s kind of rarer that new ideas are kind of, you know... you’ve just gotta be really brave and really kind of push for them. They’re often, just to an outside audience, basically inscrutable: very technical, very complicated, often require kind of large teams of people to uncover. Once they’re uncovered, they’re often easy to demonstrate, because they work. And it seems like the AI systems are not generating ideas even in domains where you can easily kind of validate that they’re good ideas. So it seems like there’s at least some more to it than just that.

Dan: I guess the question is — let’s not get derailed by this too much — but in terms of validating ideas, it’s quite complicated. Because the minute you start coming up with a nascent idea or research approach, one rational way of validating it is by thinking: how does this relate to the existing consensus body of knowledge in the field? And I think actually AIs as they exist today are, like, really bad for this. Like, I write a lot, and I’ll have, like, an opinion, and I’ll give it to Claude or ChatGPT for feedback. And they’re like: well, actually, if you look at the existing literature, then what you’ll find is... and they’ll just, like, bog it down in all of the nuance of what people have already said. And it’s like: I know that people have already said that, but I’m trying to argue for something new.

And I think this iterates into even the hard sciences. And this is why you get these, like, ferocious priority disputes. So if you think about Newton and Leibniz and calculus, right — it’s, like, this incredibly ferocious argument that ends up entangling, like, entire nations, over who gets priority, who gets the glory for being the one that made this discovery or invention, depending on which way you look at calculus. And I would argue it’s precisely that motivation — to get the glory for coming up with something new — that drives people to really back themselves, back their own ideas, where if you were just a disinterested, rational, Bayesian, truth-seeking agent, you wouldn’t. You would just defer much more to what other people have already said. But anyway — this takes us a little bit too far afield. Back to the things that we were talking about.

Fin: Actually, maybe I’ll interject very quickly there. So there’s this kind of round of random people on Twitter, like, getting, you know, their AI to generate maths results. I had a spare hour, so I kind of thought, I might as well, you know — I’ll ask my AI: just go and do a, you know, do a maths breakthrough, right? Set up a server — you know, here’s some compute, run your experiments, whatever. And it actually worked. It came up with — you know, as a total non-mathematician and dilettante — it came up with a result that seemed correct and seemed, you know, like, somewhat non-trivial. And just before — you know, I was kind of looking forward to tweeting about it and getting my, like, you know, five seconds of fame — and I, you know, asked my AI to just double-check that I haven’t been scooped. And, you know, lo and behold, it goes and checks — and I totally have been scooped, actually quite recently, by presumably some other guy, you know, using Claude Code or whatever. And I kind of wonder if you get the, like, Newton–Leibniz dynamic a lot, if you have this period where, if only you know what to prompt for — if only you know where the, you know, easy results are — then it’s very cheap to do that. And it’s basically just a kind of race for who can find the, you know, find the prompts, basically, and unlock the kind of credit they can get for it.

Dan: Yeah. I mean, definitely the way in which the use of AI is going to interact with these, like, status games that intellectuals and academics and pundits and so on play — yeah, that’s going to be interesting. Okay — this model you’ve got, Fin, of, like, what drives technological progress. As you mentioned, a lot of this is, according to your way of viewing things — if I’ve understood it correctly — fuelled by just the sheer amount of cognitive labour dedicated to research. So that’s a controversial view, as I understand it. There are some people who think — as you mentioned, there’s a vast literature on this — but they’ll think: look, like, yeah, you need this cognitive labour to get technological progress, but you also need all of this other stuff. Like people in the real world learning by doing. You need, like, fortuitous mistakes. You need particular kinds of institutions and, like, the incentives that they create. Like, merely ramping up the amount of, like, sort of pure disembodied intelligence in and of itself won’t get you huge technological gains. And my sense is you disagree with that — you think it will. So: have I fairly summarised your view? And if I have, why do you disagree with that alternative view?

Fin: Yeah. So, I mean, I should zoom out a bit and try to kind of characterise what my view is. So, I think I’m not trying to claim that, you know, the main important upshot of AI progress — or, you know, the overwhelmingly most important upshot — will be something like across-the-board technological progress driven by AI. I think more what we’re trying to do is just say: look, among the other kinds of transformative effects AI could have, here’s one that seems like, you know, we can kind of reason about, and if it were to happen, it would be a fairly big deal, kind of socially and politically. Also, I don’t think I’m trying to claim that the kind of tech progress we’ll see will be kind of general and across the board, for reasons I’ll get on to. And then finally — although it’s kind of useful to talk in these terms just to kind of give clean arguments — in practice I actually basically agree that, both historically and probably going forwards to the future, most kind of tech progress, or at least a great deal of tech or productivity kind of growth, doesn’t come from, you know, explicit research and development efforts with, you know, people in lab coats in scientific institutions. A lot of it is much messier: it’s just people in the economy in general just tinkering and experimenting and kind of sharing their ideas without labelling it as R&D. So those are some — you know I love some caveats — so those are some caveats.

Still, the question remains, right? Like: doesn’t tech progress just get bottlenecked by these kind of other, you know, complements to just raw thought — like the need to do experiments in the real world? So, one thing to say is that, you know, I think in some ways scale can, you know, eventually have qualitative effects, once you have an insane amount of kind of cognition. For example, in some cases you could begin to run simulations, whereas before they would have been so expensive or so crude as to not be worth doing. I think you could also find kind of alternative approaches, you know, in some domain, which route around the need to run experiments — where, again, they’re kind of wildly less efficient given today’s kind of factor prices, right? But in the future could be very fast.

But then the last thing to say is that I think this is just probably going to be true in at least a bunch of, you know, important tech domains, right? So, for example, if I’m trying to do drug discovery for kind of longevity or something — and, you know, I’m interested in these kind of long-term effects, and I really need to just run these, like, longitudinal studies with the same cohort over, you know, like, a decade or so — I would imagine that it is just much harder to get any kind of confident results in a domain like that, compared to a domain like maths, for example. Software is the kind of huge and obvious domain where you should expect to see major speed-ups. Certain kinds of, yeah, like, kind of predictive coding in biology, where you’ll kind of want to understand the particular mechanism — like how a particular protein binds or whatever. Yeah — presumably you’ll just see much more progress in those domains which are relatively less bottlenecked. I think it’s an open question exactly how many domains there are like that, and how well you can kind of substitute in the other ones. But I think there’s at least a kind of presumptive argument that this isn’t, like, a kind of deal-breaker on the overall thought.

Henry: So we’ve been talking in quite abstract terms so far. I think it might just be helpful to get some examples of what you see as sort of scientific projects that might sound outrageously unfeasible, but actually you might think are in nearer reach than maybe listeners, or us, would assume. So, I mean, like — what are your kind of near, medium and long-term horizons for what kind of progress this might unlock?

Fin: Yep. So the things that seem like potentially the biggest deal to me — maybe the one I would mention first is robotics. Where — you know, what dimensions matter here? One is just getting robots really cheap; another is getting them just very flexible and dexterous; and then also getting the software, the control, which I think is the most important kind of bottleneck right now. That then unlocks, I think, this kind of very general-purpose potential feedback loop, where if you can automate a very wide range of manufacturing tasks, then you start to kind of get something like, you know, self-replicating capital, which can cash out at the end on basically arbitrary physical goods, right? So the kind of cartoon way of putting this is factories building factories. Because this would be so valuable for hard power, or just economically, I would imagine that it could just attract a huge amount of, yeah, engineering effort, basically, and investment.

Then in terms of the kind of outputs — so, things like... yeah, you know, not exactly jazzed about this, but I expect military applications. For example, drones: it just seems like there’s an enormous amount of headroom for, you know, really quite powerful and quite scary capabilities there. And again, if you just look at just, you know, dollar spend on R&D, a lot of this is directed towards kind of hard-power applications. Let’s see — other things that come to mind... Okay, so drug discovery, I think, probably will be huge — I mean, again, despite the need for some physical experimentation.

Henry: I think maybe one interesting question that comes up when we start thinking about really radical technological change that could come down the pipeline is how quickly you run into some quite messy philosophical questions about it. So, like, just for example: mastery of the human body, right? Not just sort of beating diseases, but allowing us to arbitrarily alter ourselves. You know — and again, I think back to things like the Culture series, where they can do glanding, they can artificially put themselves in all sorts of psychological or physiological states. You know, maybe we could set our pleasure centres to maximum, so, you know, we can all be in these hedonic hazes. And, like, quickly you run into questions — it’s like: is this a good future or a bad future? So, I mean, that’s part of the reason I asked the question: because I think there is this kind of, like, initial shelf — low-hanging sort of fruit — of, like, things everyone wants, right? Like: yeah, cure Alzheimer’s, cure cancer, beat climate change. But then the next shelf up quickly starts to get kind of spicy, and I think there are gonna be legitimate value disagreements about, you know, which items from that shelf we wanna reach for.

Fin: Yeah. I think also one thing that’s been on my mind a bit is: there’s a kind of standard menu of, you know, somewhat science-fictional, speculative technologies that we might imagine getting, you know, by the year 2100. And, you know, on this menu there are things, yeah, like flying cars, space travel, you know, Dyson swarms, uploading, and, you know, kind of nanotech — maybe kind of medical uses of nanotech, like kind of healing kind of our brains or whatever. I think, you know, it is often possible to kind of try to dig into what would it actually take to realise these technologies. And in some cases I think it’s possible to come to a view.

So, for example — you know, citation needed — the brain is incredibly complicated. And I don’t think that we have anything near, like, even small-scale proofs of concept for, you know, nanoscale, you know, medical applications that could kind of help reverse neurodegenerative diseases, for example. That’s something people talk about. That’s just a case where — I mean, I don’t know what I’m talking about, right? I’m a total non-expert in this domain — I just don’t see any kind of shot on goal, effectively. Whereas other stories, like spreading to space and just building a ton of compute or whatever, you know, in space — in some ways I think that kind of feels similarly science-fictional. But that’s an example where — again, as a non-expert, right, I’m not an engineer, I don’t have a space background — but talking to people who do, and also just kind of using some amount of just kind of judgement... it just seems like, yeah, we just totally have all the requisite knowledge, and it’s basically an engineering challenge. Like, maybe a very tough one, but we roughly know what it would take to do it; we basically know all the physics involved.

And so it’s possible, I think, to me, that we get a world where, you know, we don’t just get the full menu at once — we get some very wild kinds of tech basically long before it’s possible to get others. Especially around — yeah, I think, you know, kind of engineering around the brain and medicine seems especially tricky to me. Life extension as well — I’m like: seems extremely hard.

Dan: I just want to have a go at summarising where we are in the conversation, and the debates that are relevant, before we move on to another really interesting part of your paper, Fin, which is about grand challenges. And just to make it concrete: so, Dario Amodei has this term that he uses — a country of geniuses in a data centre. And as I understand the argument that you’re making with Will, it’s something like: look, at some point we are gonna get something like a country of geniuses in a data centre. They might start off as just a country of smart researchers in a data centre. But because of the way in which we’re scaling compute, and increasing the efficiency of AI systems for any given level of pre-training and inference compute, it won’t stop at just a country of geniuses. There’ll be this, like, massive population explosion — at least if you’re thinking about exponential growth over the course of a decade — where you’ve got, like, many planets’ worth of geniuses in a data centre.

And then there’s a question about: if you had that, how much technological progress would that generate? And I think you and Will make a good case in the paper that it would generate a lot, even though there are gonna be bottlenecks when it comes to things like real-world experimentation and learning from doing, et cetera, et cetera. And then there are people who take the other side of that debate, where they say: look, even if you massively increased the amount of, like, disembodied cognitive labour of that kind, it’s going to be so severely bottlenecked by these other things that actually you shouldn’t expect such rapid technological progress. So that’s the debate that’s happening at one level.

But then there’s this other thing that you alluded to, which is: you might think, well, okay, once you start getting these populations of geniuses in data centres, they are going to be able to accelerate one specific kind of technological progress — which is technological progress that pertains to AI, both in terms of the software and the robotics. And then once you’ve got that, and you start getting robotics, et cetera, then all of these other bottlenecks will kind of dissolve along with them. So then you’ll get explosive technological progress. Maybe that’s an extra step — so it sort of relies on more assumptions, at least in the short term — but it’s a plausible step. And so this could all be insanely crazy, in a way that lots of people aren’t factoring in. Is that a fair summary of what’s going on, Fin, and the different positions in the debate?

Fin: Yeah, that seems right. Like, I’m trying to think about what actually is my view about what really matters most. So there’s one piece, which is: look, it seems very plausible that AI will accelerate progress in at least a bunch of just concretely important technological domains, right? So epistemic technologies; technologies involved in hard power — things like, you know, drones, and surveillance as well; technologies for spreading to space; technologies for, you know, digital technologies in some sense, for kind of, you know, economies of digital minds; medicine; and so on, right? That in itself seems like it could throw up a bunch of, you know, questions for how we kind of deal with and integrate these technologies in some way.

However, I think there’s also some story where we don’t actually see a great deal of especially novel technologies, but the world still changes dramatically. I think that could be the case if we get, like I alluded to, this kind of combination of robotics and flexible enough AI that can flexibly kind of substitute for humans, in a way that’s sufficient to start kind of autonomously scaling industry — the kind of factories-building-factories idea. So that is a world where, I think, you know, we get something like an industrial explosion, as others have called it. And I think, yeah, like I said, we already know how to build a lot of technologies which would be, at scale, really quite kind of wild — including, for example, going to space, although I’m not sure that’s the, you know, near-term most important example. And what does that take? Well, it takes enough AI progress to get the kind of flexible substitution, and it takes the robotics — but maybe not much more, other than, you know, barring political or social kind of barriers.

And then a final thing that seems really worth paying attention to is: in terms of AI progress, so far it’s mostly driven by people having ideas, and also scaling compute. There is this story that AI could automate so much of the kind of pipeline involved in getting better AI that AI progress itself — you know, however you kind of choose to measure it — inflects upwards, right? Maybe as part of a kind of feedback loop, right, where better AI makes better AI, and so on. So I don’t think that particular scenario is necessary for the others. But it seems like, if it did happen, it would, you know, bring the others forward in time, and be on its own just especially kind of hairy and kind of wild to witness.

Henry: So this is more of just a quick comment rather than a question, because I know we want to move on to other topics. But, you know, speaking as someone who identifies as techno-optimist, even techno-solutionist — I think, you know, that’s often used in a negative sense, but I think we actually have got a good track record of solving a lot of problems with technology — I think there is a failure in a lot of the kind of more optimistic discourse to connect ideas like the benefits of an intelligence explosion to the things that actually matter to everyone who’s not in our own little clique. Right? So, you know, if you say to the average person on the street, you know, here are some of the cool things that AI can do — like: I don’t want to go to space, right? You know, I don’t care about drone weapons, right? I don’t care about breakthroughs in solving the Riemann hypothesis. I care about my chronic pain. I care about the fact that, you know, my bills are high. I care about the fact that I can’t buy a house, right? So I think, you know — without wanting to go down that route too deeply — I think this is a big part of what’s currently missing in the kind of techno-optimist manifesto world: actually explaining how these breakthrough technologies can make a difference to the problems that people have right now.

Fin: Yep, I totally agree with that. I think there’s kind of, in some sense, failures on both sides — or at least both extremes — of debates around AI. You know, on one hand there’s a kind of unwillingness to recognise just how kind of radically great AI progress could be. And again — you know, you can point to specific reasons, but the more compelling reason is just this totally general, historically informed reason that technological progress, writ large, is good. Like, it generates new options and new affordances for people, right? There’s also a kind of, I think, lack of imagination about how further progress — or not just tech progress, but also just wealth — could be good. So I think there’s often this attitude of: look, I can see what I would do with twice as much wealth, right? But this kind of radically abundant world with sci-fi technologies — I’m off the train at that point. You know, surely I would just saturate. And, you know, you think about applying that to any point in history — it kind of turns out that we just fail to imagine the kind of new capabilities, new, you know, products, for example, that opened to us.

On the other hand — and I’m not ascribing this to you, Henry, or anyone in particular — but I think there’s often a kind of... there’s some attitude of swallowing that pill so fully that you just become — you know, almost as a reaction to kind of pessimistic attitudes — you just become, by default, excited about any kind of, you know, acceleration in the pace of change. Now, I think you could very reasonably be sceptical of crazy fast rates of change that, you know, various kind of Bay Area AGI-pilled people like to talk about. But if you do buy that these scenarios are possible — I think, at least, you know, my view is that the kind of appropriate reaction is to be extremely excited about the possibilities that come out of it, but... there’s a kind of missing mood or something. I think that, you know, I would feel like it’s more appropriate to be kind of not frightened, but just really quite apprehensive about those scenarios.

Dan: Okay, so let’s move on to grand challenges. So the paper makes this, I think, really kind of compelling case that even under relatively minimal assumptions — that is, you do at least have to take seriously the possibility of very powerful AI systems — but even under relatively minimal assumptions, we’re likely to get something like an intelligence explosion, which is likely to greatly accelerate the rate of technological progress. And then you can quibble with how fast that acceleration will be, and how it will happen, and so on. But there’s this other part of the paper which is then connected to this, where you reject this sort of view of transformative AI, or superintelligent AI, which I think used to be quite influential in terms of at least the way in which transformative AI gets framed. And it’s the kind of all-or-nothing view on alignment that says: look, if we manage to align advanced — and then superintelligent — AI systems with human values, well, those AI systems are going to be much smarter than we are, going to be much more capable than we are, so we can just then use those systems to solve all of our other problems. Alternatively, if we don’t align these systems, then, because they’re so much more powerful than us, they’re going to disempower us or eliminate us, and things like that.

And what you argue in the paper is: yes, alignment is a really important challenge — and kind of control, and these sorts of issues — but there are actually many grand challenges. And it’s not just as simple as: if you align it, then we can just punt all of our other problems to future superintelligence. So could you walk through that part of the essay, Fin, and then flag what some of these — I think you call them grand challenges — are?

Fin: Yeah, totally — I can try, at least. I think you put it really well. So the point is not that worries about alignment are totally ill-founded. So I think it’s clearly kind of a coherent possibility that we build AIs, and then we mess up, and then in some way they kind of, you know, cause a catastrophe, take everything from humans, it’s game over for the human race — fine. I think, you know — I kind of want to be careful about putting words in people’s mouths — but you could kind of point to a somewhat caricatured view, which says: so there is, let’s say, an alignment problem, you know, conceived of as a kind of well-described, kind of single, mostly conceptual problem, to be solved or not solved. And if it’s solved, we’ll have AIs which are kind of responsive to our, you know, intentions or instructions, in a kind of robust way. And if we get that world, then what we can do is — you know, they’re gonna be much smarter than us, and so we can ask them to solve all the rest of our problems, right? We can say: what’s the solution to ethics, or to politics, right? What should we be doing? What should we want? They can solve those conceptual questions, and they can also, you know, get to work on the kind of engineering and technological side of kind of giving us the future that we didn’t realise we needed.

Right. So, you know — the way I put it, I think I’m, you know, kinda loading the dice to make that sound like a, you know, somewhat naive idea, and maybe not many people truly believe this. But it’s, I think, worth really pressing on how this plan does not seem especially robust, even if we do end up with AI systems that are very sophisticated at thinking about the kinds of questions we’re interested in solving. One way of putting this is: just think about politics, like, historically and today, and kind of other social problems. To some extent these are problems because we’re confused — we’re conceptually confused, right? About what’s the kind of solution; how exactly, you know, is this kind of problem arising; are we missing concepts here? But for the most part, problems of politics are not problems which get solved with, you know, a very smart person coming up with a solution, right? I think, you know, there’s no special reason why politics as such will just end when we get superintelligence.

And then, more to the point, I think that you can imagine scenarios, for example, where we have AIs which are, you know, in some sense intent-aligned — they do what they’re told. But, for example, there’s a small number of actors in the world who, you know, get to tell the large majority of AIs what to do, right? They just hold a great deal of power, and that just presents all the problems of power imbalances that we’re familiar with through history — in some ways, they could be much worse. So that’s one problem: this kind of extreme concentration-of-power worry, which I can kind of elaborate on. But there are other problems too — or, you know, at least challenges. So one is: let’s say that there are ways to get really great outcomes in some kind of impartial sense, and maybe the AIs can tell us about them and help us reason through — but they’re just unpopular, right? Or people are not especially motivated to act on them. That would be one thing. And we can look again to history, where, at least from our vantage point, it’s pretty obvious that societies from the past have kind of engaged in these, you know, morally embarrassing practices, right? In some ways, that wasn’t for lack of a certain kind of awareness.

Yeah — and I think, you know, you can kind of list more specific challenges. So you could also imagine, in the extreme, just one very powerful actor chooses to lock in what they care about at a time. So they use kind of technological means to just seal themselves off from any further kind of, you know, change or evolution — which are, you know, processes we’re just used to today. You could imagine just kind of bad equilibria — kind of just failures of coordination. You know, again: even today we know what coordination failures are. We can write them down on whiteboards, right? Describe them very well. We still get them. So, you know — I realise this is quite kind of abstract, and I could kind of try to be more concrete if you want — but this is the kind of general story where just getting AIs that do what we tell them to do — even very wise AIs, or very smart AIs — could just easily not be enough to get the just truly great futures that we could get.

Dan: I mean, you list a number of grand challenges in the paper — you just alluded to some of them. Like, for example: AI takeover; highly destructive technologies; as you mentioned, value lock-in. Obviously we can’t cover all of these now. There are two that I’m especially interested in discussing — I think Henry is as well. One, which most people don’t immediately think of in this area, has to do with space and space governance. And the other, which touches on these sort of incredibly difficult ethical questions that are going to emerge even once we’ve got potentially superintelligent AI systems that are aligned — which is just this explosion in the number of digital minds, and more broadly kind of AI-based intelligences, and how we should think about those, and what an ethical, just society where we’ve got such systems — what that might look like. Let’s start with space governance — something I know nothing about, but I think Henry’s interested. What’s the connection here at all? Like, what’s the connection between — okay, we’re gonna build superintelligent systems — and space?

Fin: Yeah. So I think the general story would be: there is this kind of, you know, kind of worldview where you can imagine we get this kind of inflection in AI capabilities potentially quite soon, right? As a result, we get this kind of period of rapid technological change and progress, and just across-the-board growth — including industrial growth. That’s the kind of background, right? You might not buy that possibility, but you might think it’s plausible. Then you can ask: okay, on that possibility, just what else pops out as looking like, you know, a more near-term issue than you might previously have thought? Well, space comes to mind, right? So space is very big, and so you can do lots of stuff in it. Eventually there could be — in some handwavey sense — more stuff happening in space than is happening on Earth. We have a rough sense of what it would take, as a matter of engineering, to do a bunch of stuff in space — like, we’re already doing stuff, mostly in orbit. So again, on this kind of world of tech progress and industrial scale-up, potentially just vastly more stuff starts happening in space.

And then finally, it’s kind of amazingly under— how would I put this? — it’s very thin on institutions and laws and regulation, for better or worse. Partly because until fairly recently there hasn’t been a great deal of interest in figuring out, you know: how do we govern space? Partly because mostly it’s just been a matter of states, and it’s quite hard to get agreements between, you know, great powers. But if you just buy that a bunch of, like, stuff could happen in space — I can say more about what that means — you know, in, like, a matter of decades, rather than by the end of the century or so, it could be that we’re in this kind of period now where there’s a bunch of plasticity and kind of openness about what kind of institutions we kind of negotiate — which could then be really quite influential for kind of determining how things go.

Henry: So — just throwing my thoughts in here, because I’d love your feedback on this. I know, Fin, you’ve recently written a report on data centres in space. So I think two things that possibly are worth bearing in mind, when thinking about the potential for space to take off really, really fast in the next decade, are: firstly, there are economies of scale, when it comes to launch and construction in space, that we’re only really beginning to exploit. We’ve really seen this sort of— with SpaceX’s Falcon rocket, you know, launch costs have come down really far already. And we’re still talking about two or three launches a month of the Falcon rocket — and already that’s reduced costs massively. You know, if we move up to 30 launches a month, 100 launches a month — again, I think we’re gonna see bigger and bigger economies of scale. Launch costs could plummet. And equally, of course, data centres in space — this creates a pretty compelling use case, a pretty compelling justification, for putting things in space.

And equally, you know, one of the big obstacles to doing things in space, of course, has been that humans are spectacularly ill-suited to being in any environment outside of Earth. And if you try and do stuff even moderately far from Earth, right, you run into real latency issues in, like, controlling things remotely, for example. So once we have fully autonomous intelligences — you know, we could just send off an intelligence... I mean, this is a crude example, right? But let’s just say you wanted to build a factory on the Moon. Once you have reliable autonomous robots: dump them on the Moon and say, you know, build some solar panels, you know, build outwards, build some factories to start making either data centres or... beginning the Dyson swarm, right? This stuff could actually take off pretty fast, because, A, of the economies of scale, and, B, because some of the constraints don’t apply. Right — so this is how I get my spacepunk future that I’ve been looking forward to since I was a kid. I’m just curious, like: what do you see as the main obstacles to that? How realistic is it?

Fin: Yeah — so, I mean, I totally agree with what you said. One thing to, yeah, really press on is: I think futures in space which look like Star Trek, right — where it’s a bunch of humans in, like, the kind of big O’Neill cylinders, or space habitats, and they’re, like, orbiting around the Sun, or maybe they’re going on, like, you know, space missions to other stars, and literally humans are kind of spreading across space—

Henry: Seeking out new life and new civilisations.

Fin: Quite right. That does not strike me as especially plausible, or easy to imagine. Well — I mean, it could happen, right? But other things could happen so much faster, and in some sense just overtake those processes, right? It’s just wildly kind of inefficient to bring along this kind of, you know, sack of biological human, where you can do it kind of so much more efficiently. So — not imagining Star Wars. But, yeah — what seems, like, actually plausible? So one is, like you said, data centres in space — at least in the kind of medium to long run — could be quite appealing. The simple reason is there’s loads of solar energy in space, and if launch is very cheap, then you might as well throw them up there. There are, yeah, other uses of orbit — so there’s kind of telecoms, and there’s remote sensing.

Stuff on the Moon — so I’m, like, a bit sceptical of a lot of the things people say. So, for example, you know, mining the Moon for, like, nuclear fusion — like, helium-3 — as far as I can tell, doesn’t kind of pan out at all. Similarly, mining asteroids and then, like, throwing rocks back to Earth: there’s just so much, you know, material on Earth — I don’t really see the case for that. The thing that does seem, you know, just very technologically feasible to do is mining and manufacturing in situ with planetary bodies around the Sun. Where — I mean, look, this is, like, really quite far out at this point, at least in terms of the, you know, the kind of engineering required — so I’m not claiming we should kind of expect this to happen in a few years. But you can imagine, if you can kind of build, as it were, like, a seed — so it has some kind of initial robotics, and instructions to set up mining infrastructure, and the kind of infrastructure required to produce the next few seeds — then you have just this kind of self-growing process, and then you can kind of, again, convert that into whatever you want. You know, that seems like a pretty big deal once it becomes possible. Again, I think it’s just—

Henry: Yeah — the von Neumann, sort of Factorio, exponential growth of manufacturing capability in situ.

Dan: I have two thoughts that conflict with each other when I think about this. The first is: this seems hugely important, and we need more people thinking about this in an informed, rational, forward-looking way. Because at the moment, my sense is very, very few people are thinking about this — especially if you’re thinking about, you know, institutions and regulations and laws and so on. Very few relative to the scale of the challenges and the opportunities. Then the other thought I have is: this stuff is so weird, and it’s so, like, out of sample relative to what we’ve experienced in the past, that, like... can we think about this in a sensible way that would improve on just muddling through?

Like, if we go back 500 years: the average human being is living in this, like, micro-world. They’ve got, like, a really tiny community. There is this whole planet out there of, like, different peoples, but they know basically, like, very little of that. And there’s certainly no, like, global governance regimes and so on. And if you really try to think about it, you know — 500 years ago, go back 3,000 years ago — what would a world where you’ve got these huge things called nation states, embedded in the international regulatory apparatus — what would that look like? What should that look like? It’s just not obvious they’d make any progress on that. And now, when we’re thinking about space exploration, and people making use of, like, superintelligent machines and autonomous robots and so on to start exploring — what confidence should we have that really anything we can think of now is going to make much of a positive difference to how that’s likely to unfold?

Fin: Yeah — I think that’s, I mean, just, like, an extremely reasonable question. And, I mean, roughly for the reasons you give... So — you don’t think that this should be kind of top of the list of priorities for people to, you know, kind of bash their heads against and kind of work on, when there are so many just, you know, already quite imminent and real problems in the world. That said — I don’t know exactly how to articulate this, but, you know, I feel like we — as in, you know, humanity writ large — have accumulated some lessons about how to kind of arrange ourselves and build institutions that kind of broadly work. And I think a lot of the lessons that we’ve learned are quite robust — that is, they don’t depend on details, right? So you think about the kind of innovations of kind of liberal political thought: about the idea of rule of law, and, like, independent judiciary, and democratic mechanisms, and market mechanisms, and kind of economic concepts around, you know, efficiency and how markets work. You know, those don’t depend on, or mention, technological details. And in fact, you know, the past century has seen kind of, in some sense, wild amounts of technological change, but a lot of those ideas have, you know, really just proven quite robust.

And so I think, you know, at least we have this kind of null hypothesis, right? Which is: if it’s really hard to reason about how specific technological changes kind of change the game board, at least we kind of know there are certain arrangements which are just better than others. And that seems quite robust. So, for example, you know: arrangements where there is just no rule of law, and there’s some new resource, and there’s nothing better than just racing to grab the resource because no one’s gonna stop you — that just generally seems a bit hairier, and a bit worse, than, you know, building some institutions which kind of allow people to, you know, peacefully kind of negotiate ownership of these resources, right? And I would say at least we can do those kinds of things about space — which is, we can punt all of the details, a lot of the implementation details, to when we know more. In fact, we should. But at a high level, like, maybe we can start advocating for, you know: let’s just have some mechanisms to, like, you know, have these kind of peaceful alternatives to conflict, or to racing, for example.

It’s very tempting, I think, to try to, from scratch, like, devise new kind of mechanisms or institutions that are, like, perfectly fit to, you know, sci-fi technologies — and I think often we do need to at least extend the kinds of arrangements we have. But I think at least we can do that, right? I don’t think we need to — indeed, I think we often can’t — guess about how the details pan out. But those are things that we can punt.

Henry: So, as Dan mentioned earlier, another really big issue it’s worth drilling in on — perennial topic on the show — AI consciousness, AI welfare. So: you have some quite spicy views on this, right? You have quite a deflationary view of consciousness. Do you wanna tell us about it?

Dan: Actually, Fin, just before you do — I just want to frame it in terms of what we’ve already discussed. In terms of: okay, we get rapid progress in AI; this is going to trigger lots of technological progress; this is all throwing up grand challenges, and many of these challenges can’t just be reduced to — in fact, most of them can’t just be reduced to — aligning AI systems. And one set of challenges here is just: we are going to be building huge numbers of artificial intelligences — both kind of disembodied digital minds and, very plausibly as well, lots of embodied, autonomous, semi-autonomous robots. And that then raises huge ethical and political questions, like: how should we treat these systems? How should we live together with them? And I think many people think the first question you should ask, for any of those other questions, is just: are these systems actually conscious — in the sense of: are there lights on inside? Do they have phenomenal consciousness, as many philosophers would put it? And so you flagged this sort of question of how to treat and interact with digital minds as a grand challenge in your paper. You’ve also got this really interesting — and I think very persuasive — essay where you kind of give a deflationary answer to that question concerning AI consciousness. So: what is that deflationary answer? And do you think it’s relevant, then, to how we should think about these big questions about digital welfare and AI rights and so on?

Fin: Yeah. So I think my view is relevant, but it makes me just feel very confused, and not sure how to approach those downstream questions. So I sometimes think about analogies to biological life, right? So, you know, I think sometime around, let’s say, kind of the middle of the eighteenth century, the kind of prevailing view on what distinguished living things from non-living things — that is, you know, plants and mice from rocks and chairs — is some kind of distinctive, you know, life force that was shared by all and only living things, right? And had certain kinds of properties that meant that there were kind of discoverable facts of the matter about which things were living. Maybe it was imbued by God; maybe, you know, there’s something else going on. Now, it turns out — so we did a bunch of biology; we learned a huge amount about how living systems work. And in the course of doing that, one thing we learned is that there is no such life force, right? There’s just a collection of mechanisms that we associate with life. And there are also fuzzy boundaries to the kind of life concept — well, there’s an ambiguity about which kind of systems you might treat as living. Viruses, for example, right? And so, you know, you can ask this question: is a virus alive? If you were a kind of, you know, scientist in 1760, you would say, you know: we just need to go and discover that — you know, or it depends, kind of, what God chose. Now, what we say is: well, just tell me what you mean by alive. Nothing much else hinges on that, other than your choice of definition, right?

As a side note, by the way, I’ll say: I actually looked into, like, this life force thing and how real it was. My sense is it’s actually much more complicated — like, the history of science is just much more all over the place. But it’s a very convenient myth, so I’ll stick with it.

So, by analogy, I think that something similar might be going on with this kind of concept of consciousness — or, you know, phenomenal consciousness, you might say, right, in particular. Where it seems that people share an intuition — at least people who have decided to kind of, you know, read philosophy about consciousness — that there is a kind of discoverable and non-controversial, or let’s say non-ambiguous, fact of the matter about which systems are conscious. That is: you take some system; it might superficially appear conscious; but you can still ask, are the lights actually on inside, right? Is there something it’s like to be this chatbot? Or to be this mouse, right? The analogy to life would suggest that things are more complicated, and that really there are not facts of the matter — at least in this kind of sharp and non-ambiguous way.

Why might this be true? Well, there’s this kind of view on consciousness that’s often called illusionism — or strong illusionism — about consciousness, which says: you know, this thing that we have the impression exists, this phenomenal consciousness thing that has this kind of bundle of properties — like, it’s irreducible, it’s in some ways incommunicable, it’s kind of strictly always present or not present — this thing just is not real. So we only have impressions of it, but those are mistaken impressions in some way — in the same way that this life force thing that we think exists only appears to exist but doesn’t in fact exist. That doesn’t mean that consciousness in general — tout court is a phrase that I picked up from yesterday — is totally meaningless, right? So clearly we can ask interesting scientific questions about various phenomena we associate with phenomenal consciousness. We can also ask questions about which systems, you know, have beliefs about whether they are in fact conscious, right? So I think it’s clearly true that both of you and myself believe that we are kind of phenomenally conscious in some way. But we just can’t ask well-posed scientific questions about which systems are in fact phenomenally conscious — or, rather, we can, and the answer is: no systems.

And so that’s kind of where I think — at least where I’ve ended up, having tried to think about this a little bit. I think the arguments are fairly compelling. I think the result is just a lot of confusion about how to answer the downstream questions about: okay, what do we actually do? You know, we have all these ethical questions — how do we answer them now? But that’s where I’m at.

Henry: So I do want to take this to questions about the implications of this view for ethics. But just to sort of go through some of the kind of standard dialectical moves when illusionism comes up. So, you know, this is obviously an increasingly mainstream view, I think, in philosophy of mind and consciousness studies: that consciousness in some sense doesn’t exist — or at least doesn’t have the kind of properties we associate with it. This idea of the élan vital as a historical analogy — the life force. So the standard response to that from the anti-illusionist camp is: look, the reason we posited a life force to begin with was to explain certain kinds of structures and behaviours, right? So, you know: why is there this class of things that can reproduce, grow, and have all these properties, and then all these other things, like rocks and stones, that don’t have these properties? We need to come up with a mechanism to explain this difference in behaviour. Consciousness, according to the anti-illusionist camp, is fundamentally different, right? Because we’re not invoking it to explain behaviours, right? We’re invoking it to explain a manifest phenomenon that we are all immediately acquainted with, right? Now, the illusionist can say: well, you know, that’s sort of begging the question, right? If my view is correct, then there is no manifest phenomenon. But I think that’s a very hard pill for people to swallow. You know, there’s this kind of standard interior pointing move that says: but come on, this — this, inside my head — as if you can point inwardly. That can’t be an illusion. There’s something real there. So I do think there is an interesting disanalogy there with the kind of élan vital move — that, you know, we’re not trying to explain behaviour. So, like, you know, you’re welcome to give your reflections on that.

But I also wanted to say that, you know, one reason I think it’s quite hard to at least get rid of consciousness, without just smuggling it back in under a new name, is that it seems to have this kind of direct connection to value. You know — so if I’m gonna go for an invasive piece of surgery, and, you know, I’m about to be given a new type of anaesthetic, and I say to the doctor, you know: will this render me unconscious during the operation? And the doctor says: well, you know, it’ll reduce your attentional faculties, it’ll knock out your working memory, it’ll reduce your social cognition... It’s like: no, no, no, doc — I wanna know, am I gonna be conscious, right? This is the property that matters to me, right? And in the same way — if I’m talking about shrimp, right? If I wanna say: do I wanna allocate money to shrimp welfare, right? You know, you can say: well, you know, here are some interesting facts about shrimp behaviour and shrimp social complexity and shrimp intelligence... It’s like: that’s, you know, maybe interesting, but that’s ultimately not what matters morally. What matters morally is: are there lights on inside, and do those lights hurt or feel good? If so, they are welfare subjects; if not, they’re not. So, like, every illusionist attempt I’ve seen to try and sort of grapple with this problem says: well, okay, you know, maybe consciousness isn’t, you know, what matters, but this quasi-consciousness, or something like that — you end up reinventing this notion of consciousness. So, I don’t know — do you have any thoughts on this? Am I being harsh on the illusionist response?

Fin: Yeah, totally. So, I mean, yeah, I think two points there. I guess the first, you’re saying, is: unlike this analogy to biological life, the thing that phenomenal consciousness is invoked as a concept to explain — it’s not this kind of appearance, you know, of just various behaviours; it’s this kind of manifest, obvious, first-personal reality that I am conscious right now, right? And so one response is: well, I can explain why you say that without ever invoking phenomenal consciousness, right? Because I can talk about how your brain works. I think that is not enough, because you might just say: look, I’m not trying to persuade you of anything — I’m just saying I know, right? I appreciate I can’t tell you; but, you know, anyone, you know, can just kind of introspect and appreciate that they are conscious, right? Then the response has to be — I often think this is just where the argument ends, because there’s a kind of just appeal to brute fact, right? For better or worse.

One potential response is: so, the illusionist is not trying to kind of eliminate beliefs, or judgements, or, you know, kind of epistemic concepts. So — I have, kind of, again, reflecting internally — so, not trying to persuade you, but I’m thinking — so, I certainly have a belief that I’m phenomenally conscious, and that’s non-mysterious, so I can help myself to that, even if I’m an illusionist. What more is there to explain? I think that at least starts to feel very confusing.

Henry: But I mean — just very quickly — you know, a base-model LLM will believe it’s conscious, right? Yep, presumably. And so you might say: look, you know, you believe you’re conscious; so does LaMDA; so does GPT-3 in its untrained form. We can just give explanations for why both of you think that — problem solved. But clearly there is a disanalogy in why we believe we’re conscious. You know, GPT-3 or whatever believes that it’s conscious for some really bad reasons. At the very least, my reasons for believing I’m conscious are very different.

Fin: Yeah — I mean, it’s very hard to know what to say. So, I mean, obviously I share your intuitions, because I’m human, right? So I have this intuition that there is something extra. What I’m saying is that if my kind of criterion for, like, submitting this as an argument is that I can articulate it in any more detail than just “this seems wrong” — that is, without just begging the question — I really struggle to know what to say, right? So, you know, it’s conceivable that I actually just can’t articulate to you what is the difference between me and some AI system that, you know, merely has the kind of, you know, epistemic stances associated with believing that you’re conscious. So, look — there’s not much more to say on this, or at least I don’t know what more to say on this. I’m sure other people do have more to say.

Henry: I mean, I think there is, like, several shows’ worth more to say on this. But with that in mind, maybe we should move on to the ethics side, which I’m also really curious to hear your thoughts on.

Dan: Well — actually, before we move on to the ethics side, could I just sort of chip in to try to summarise, especially for people who aren’t in the weeds in the kind of philosophy of consciousness literature. So, Fin, I take it, as someone with your kind of illusionist view, they’re gonna say: look, we can give functional, behavioural analyses of different psychological states and processes. So we can talk about: this system can perceive, it can imagine, it can deliberate, it can feel pain even — in the following sense — and then you would cash that out in terms of the capabilities, the propensities, of an information-processing mechanism. So there’s no issue with saying that a system has psychological states, including those that we typically think of as being connected to ethics, in that sense. Now, there are some people who then want to say: okay, it can perceive and imagine and deliberate and plan and act and feel, in those functionally, behaviourally defined ways — but I’ve got this extra question, which is: are the lights on inside? And your view, as I understand it, is just to say: well, there’s nothing else to explain. There’s no extra property of the universe. We can still care about whether a system can perceive and feel and act and so on. But once we’ve cashed that out functionally and behaviourally, in kind of usual, standard scientific terms, there’s no extra deep metaphysical question about whether the lights are on inside.

But there is this question of why animals like us are disposed to say that there’s an extra question — why we’re disposed to say: are the lights on inside? And if we can offer a convincing explanation of why we’re disposed to say that, that doesn’t itself appeal to there actually being some extra property, then we’ve done everything we need to do from the perspective of a kind of satisfying scientific worldview. Is that a summary of your position?

Fin: Yeah — I think that’s... you said it better than I could have said it, again. What can I add to that? It’s useful to think about other cases of debunking. So if someone comes to me and says, “You’ll never believe it — I saw a UFO last night,” then I can ask: what grounds do I have to believe that there was actually a UFO, right? What I can do is — let’s say I, you know, have access to their brain state — I could explain why they said, in the moment, that they saw a UFO, without ever kind of using the word UFO, right? So, you know: neurons fired in their brain caused them to make those sounds. That is actually not enough to debunk whether or not there was a UFO, right? Just commonsensically. However, suppose that I, you know, checked the news: there was a weather balloon, right, and it had a light on that looked like a UFO. Then I can debunk whether it was really a UFO, because I just never have to invoke anything like a UFO — even if I’m trying to conceal it in, you know, kind of other language. So that would be the claim with illusionism.

And then I think there is, like you said, something that the illusionist still has to explain — which is, I think, so far not explained, or at least there’s no consensus on what the story is — which is: if consciousness is an illusion, it’s super weird that basically everyone who thinks about consciousness ends up with the same illusion, the same impressions. And basically no one seems to be able to convince themselves out of it — unlike virtually every other, you know, perceptual illusion or kind of epistemic mistake that people can, you know, tend to find themselves in, but then persuade themselves out of. Like, you know, imagine a visual illusion where you can just kind of look at it another way and realise you’re wrong. So that remains to be explained. So illusionism is not, I think, to date, a full explanation of what the hell’s going on with consciousness. But it’s a kind of proviso, or a kind of: look, here’s the direction that seems most likely to pan out — which just denies that consciousness is real at all. Yeah.

Dan: I think it’s worth just saying two things that make me very sympathetic to this view, which have mostly just been implicit in what we’ve been saying, but are worth kind of making more explicit. One is, like: how compelling this view seems partly depends on how you frame it. So if you say “consciousness is an illusion”, that sounds bizarre. But really the view is not that consciousness is an illusion, but that it’s an illusion that there are these intrinsic, ineffable properties that elude functional, behavioural explanation. And if you put it like that, it starts to seem, I think, much more reasonable.

And then, second: in the background here is just the idea that, like, this seems continuous with ordinary scientific investigation, of a kind that has been basically the only source of knowledge about the world we’ve ever produced. And to the extent that the illusionist position seems much more coherent with that kind of scientific epistemology, that’s, like, a really, really strong point in its favour — which should be relevant to this debate you two are having about, like, who’s begging the question against who, and which intuitions are we relying on. If you just step back and ask, like, which position endorsed here is more aligned with a general scientific attitude to these questions — I don’t know, my sense is clearly it’s this illusionist position. But my sense is, Henry, you disagree with that, at least.

Henry: Well, here’s sort of one question I have — a kind of meta-question for the illusionist — which is: is this a view you’ll ever be able to convince people of? And if the answer is no, then it’s almost — okay, maybe that’s not irrelevant, right? But I do think this question of how do we achieve something like consensus in this domain is a question that, you know, the non-illusionist might have an answer to. Like, if you’re just a non-illusionist reductionist, you might say: look, we are gonna figure out what consciousness is. It’s gonna be amazing. It’s gonna be the best theory ever. Maybe we need superintelligence to help us figure it out, but we’ll be able to pin it down, and we’ll be able to see which animals have it, which animals don’t. We’ll be able to salvage a lot of our intuitions. It’ll be nice and clean. Everyone will be convinced. You know, they’ll come up with a new Nobel Prize just for philosophy, just to give to whoever comes up with it. But, like, the illusionist story — I’m not sort of sure what happens. It’s like: you just get progressively more convincing arguments, and, like, fewer and fewer people believe in it over time, and just gradually it falls out of the discourse. I mean, maybe — maybe that would happen. But, yeah — I mean, now, that’s not an argument against the truth of illusionism, but I think in some ways it would be much messier and worse if something like illusionism was true. I also have my other first-order arguments — or first-order disagreements — with illusionism, but maybe I’ll save those for another time.

Fin: Yep — okay. Yeah, I’ll be curious to chat about that at some point. So, I mean, I agree and disagree with what you said, Dan. I think that it just clearly is very hard to internalise. So, you know, I would say that I’m very sympathetic towards illusionism, although not fully confident. I still just can’t kind of wrangle my intuitions to get on board, right? This is kind of the System 2 part of my brain that’s doing all the work. And I think it’s not possible to sympathetically reframe illusionism to make it sound, you know, kind of obvious after all. Like, you really are just denying something that is kind of just pervasively and strongly kind of intuitive. One analogy might be to the sense that people have of a kind of continuous and distinctive self — where I might think that there is always an unambiguous fact of the matter about who is me at a given time, as long as I’m still alive. So that’s an example where I think philosophy — philosophers — have done a good job showing that those intuitions are false. And I find those arguments moving. And, you know, the day after I read those arguments, I’m like — my intuitions haven’t changed, right? Except when I kind of go and kind of read about that stuff again. I think that shows—

Henry: Hold on, hold on — you don’t feel like you’re living in the open air, as Parfit put it, you know, once you get rid— Yeah. Yeah, exactly.

Fin: Right, right — my glass tunnel. Yeah. I haven’t read that passage enough to internalise it. But, you know, it shows, I think, that part of the job of philosophy is to try to kind of challenge pervasive intuitions, even when it doesn’t succeed in kind of actually undermining them in any lasting way. But then the second thing you said, Dan, I think I agree with — which is that illusionism is in some sense the view that consciousness goes the way of every other kind of empirical or philosophical mystery historically, which is that it gets in some way dissolved by just standard, you know, conceptual or empirical methods, right? So it would be exceptional in that sense — uniquely exceptional — if we needed to kind of devise new methods to understand it.

Now — and then, I guess, going back to Henry’s point earlier, which was: well, hang on, here’s a kind of, you know, huge problem, which is: we base so much of our thinking about ethics on consciousness concepts, right? So I care about helping, you know, other people, or non-human animals, you know, in some cases exactly because I think that they’re capable of suffering, or feeling, you know, joy — and I, you know, want to avoid the former and, you know, promote the latter, or whatever. And so — I mean, here’s some reactions, right? One reaction is that — and I read Henry as saying this — you can use this as an argument against illusionism. Which is: I think my kind of ethical intuitions are true, and they imply consciousness — or the kind of phenomenal consciousness required to kind of ground those intuitions, right? Therefore, we have to help ourselves to those consciousness concepts. I do think that’s a good argument.

Henry: Or maybe the kind of Moorean version of this is: I’m more confident that there is an ethically salient difference between animals and rocks, grounded in a specific kind of internal property — I’m more confident in that ethically salient distinction than I am that any clever philosophy-of-mind argument to the contrary is false.

Dan: Moorean in the sense of G. E. Moore, who used that style of argument to dismiss certain kinds of arguments.

Henry: Thank you — thank you, Dan. Sorry, yeah.

Fin: Good stuff — thank you, Dan. So — how to react to this. One thing to say is that this might be a bit cheap, so I’m kind of curious for your reaction. When I think about kind of ethical attitudes, especially historically, they don’t seem to me grounded explicitly in kind of phenomenal-consciousness-related concepts. Or at least they don’t always seem to be. Sometimes they seem to be grounded in concepts which I think are confused, right? So maybe you might care about helping someone to save their soul. If you think there are no souls, the fact that people had kind of soul-saving intuitions is not an especially compelling reason to believe in souls. But also you can kind of ground ethical intuitions in terms of things like, you know, fairness, or objective goods, or preferences. So it’s not clear that you’re totally at sea. But then, finally, even if you were — even if you thought, my god, like, I have no idea what kind of, you know, foothold I have here — I don’t know... maybe I’m, like, a bit sceptical of the kind of Moorean-type arguments in general. I’m like: well, maybe that’s a bullet you have to bite. It seems possible to me.

Henry: So I guess one point you could make on this — that I would expect you to be sort of sympathetic to, Fin — is that human moral psychology is just a really messy grab bag of System 1–type responses that we’ve evolved over the years, mixing everything from disgust and basic survival-related feelings and intuitions to all sorts of more contractarian, or sort of social-dynamical, intuitions. And, like, one of the achievements of twentieth-century politics and ethics is realising that some of these are probably less well-founded than others. And that, you know, the idea that I have a greater moral obligation towards someone just because they’re in my in-group is maybe not a great way to think about things. And, more generally, you know, the broader idea of the expanding moral circle, right — which, again, I expect all of us are to some degree sympathetic to.

And this has been grounded primarily on a basically sentientist framework. Maybe not essentially, right? To be clear, I don’t think you can’t have an expanding moral circle without being a sentientist. But, like, Peter Singer is a sentientist, as are many of the loudest advocates of expanding the moral circle — and particularly, you know, when we think about non-human animals and so forth, right? There is this kind of sentientist idea in the background. So I would say that the move away from, you know, moral obligations grounded in souls and honour and piety, towards a more harm-based framework, has been a real form of moral progress. And illusionism threatens to kick away, you know, the last supporting pillar there.

Fin: Yeah. I mean — so, like, to be clear: my actual view on what the hell this means for ethics just is overwhelmingly a view of being, like, very confused, and, like, slightly worried, for the reasons you say. So, I mean, some things to say here. One is: imagine that you have a faith, and you’re questioning your faith, and you’re worried that, you know, on the other side is this kind of moral void, right? If God is dead, everything is permitted, and so on. I think — I don’t know if either of you are God-fearing men — but if you are an atheist, I think most people have this attitude of: it turns out that, you know, you still just care ethically about other people, right? And if you kind of originally thought that your ethical attitudes were grounded in belief in God in some way, and then you lose your faith, you can kind of either change your motivations, or realise that, after all, your motivations weren’t grounded in that thing you thought they were.

I think that kind of goes on in my head, where I think: okay, what if it turns out that phenomenal consciousness is not real? So I previously did think that the most kind of compelling way to ground my ethical intuitions, you know, in a kind of systematic and fair way, is in terms of facts about how much pleasure and pain there is in the world. Now, you know, if illusionism is true, I can’t help myself to those facts. I don’t feel like I therefore have to give up on the intuitions I have — that it’s good to help other beings, and, you know, to prevent needless suffering, for example. Yeah — I don’t feel like I need to kind of revise those attitudes. It feels way more kind of plausible to me that those attitudes just don’t really necessarily depend on other metaphysical beliefs. Now — you know, exactly what do I do? Like, how do you kind of cash out ethical views in, like, a systematic way? I have no idea. But my sense is that there’s no need to panic, if that makes sense.

Henry: So — two very quick reflections on this, and then, you know, I’ll pass back to you, Dan — or Fin, if you’d like to respond. So I think there is this kind of viable project, that you’ve seen some people start to sketch out, where, you know, we can take our intuitions around phenomenal consciousness and maybe salvage some of them, maybe lose others. You know: maybe rather than saying this pain matters because it’s conscious, we say this pain matters because it’s globally accessible, or this pain matters because it’s the target of a higher-order thought. And even as I say that, right, I think the shape of the worry starts to come through — which is: it’s not clear that any other properties in the brain can carry that same sort of foundational role as phenomenality. There’s just an obvious wrongness — one that’s very hard to vocalise — about the badness I experience when I am in conscious pain. And if I say, look: negative reward signals that are globally broadcast are intrinsically bad... that suddenly starts to look really mysterious to me, and much weaker. Go on, Dan, yeah.

Dan: Can I just double-click on that? So presumably the view would not be, you know: pain is bad because it involves global workspace architecture under such-and-such conditions. Presumably the view is: pain is bad. What is pain? Well, pain is the following. You’re not giving a kind of justification-based argument, where you’re saying pain is bad because of this other thing. You’re just giving an account of what pain is. And then there’s a question, which is: well, why is pain bad? But that’s going to be a question for anyone. I don’t get why that’s a distinctive question for the illusionist. What am I missing with that?

Henry: Well, I mean, I think part of the answer that the non-illusionist can give is that pain is bad because of its phenomenal character, right? The felt unpleasantness of pain is what makes pain bad. And if you come along and say, well, that phenomenal character doesn’t exist, then it’s just not clear what you’re gonna ground that badness in.

Dan: Yeah — I worry that there’s just sort of talking in circles there, though. Why is pain bad? Because it involves suffering. Why is suffering bad? Well, it’s, like, a negative experience. Ultimately, I think you’re just going to be able to cash these things out functionally and behaviourally. Then there is a question about, like: why is anything bad? And that’s just, I think, not going to have a satisfying answer — because I’m a nihilist, for independent reasons. So I don’t think anything is going to ground our moral judgements. So, like, in some sense it is all just a kind of make-believe that we are motivated to play, as a certain kind of pro-social ape that’s been encultured in a particular way. So, like, there are complicated, like, meta-ethical questions. But I just don’t get why I should be that bothered by the implications of illusionism specifically. Like, it’s just giving an analysis of what pain is. It’s not, in and of itself, trying to give you an answer to the question of why pain is bad.

Henry: So I would agree that on a non-illusionist position there is still this question of, well, what is it about the phenomenal character of pain that’s so bad, right? But the non-illusionist is not claiming to have explained this whole situation, right? Whereas the illusionist, I think, is already more confident in an answer than the non-illusionist — namely, that this phenomenal property, this kind of thing, doesn’t exist. And just to offer a historical analogy here again, right? So, you know, the illusionist can appeal to the élan vital. Here’s another analogy you could appeal to, you know: if you read ancient Greek philosophy, it’s full of people like Parmenides and Zeno, who confidently assert that, like, motion is impossible, or that there’s only one object. And they come to all these really radical, well-argued logical conclusions, right? And the answer, usually, to them is often things like: well, they didn’t have convergent series back then, right? They didn’t have the maths to understand this; or, like, physics was just at such a primitive level that they weren’t in a position to realise that these things that they thought somehow were logically impossible were actually broadly resolvable within a broader theory of physics.

And I think we know so little about the brain, right, that the illusionist’s call that, like, there can never be anything like this — this clearly just doesn’t exist — is premature, right? Our knowledge of neuroscience — neuroscience is such an infant field, right? In the sense that, like, we still don’t have any really good models of how just basic stuff, like representation, works in most of the brain. And so I think to, you know, to give up and say, look, there can never be anything that actually satisfies most of our intuitions about this — might well be premature. And as we start to understand more about how coding works in the brain, how we create this kind of unified psychology, right — something might well emerge that actually is a target for some kind of reasonable identification with at least enough of our pre-theoretical ideas about consciousness, for us to say we found consciousness, rather than consciousness doesn’t exist.

Fin: I’m not sure how persuasive I find that. So — maybe I’m misreading you. So suppose that we found some kind of brain process that just perfectly correlates with our intuitions about kind of whether and when phenomenal consciousness is present. I agree that maybe that deserves just the name “consciousness”. I think it’d also be, in some sense, good news, right? Because we don’t have to confront these ambiguous or edge cases, where, you know, some kind of associated properties are present but others aren’t. I think, nonetheless, you could still maintain an illusionist view — which is: we remain metaphysically confused about consciousness, phenomenal consciousness. That is, we are confused, for example, that it’s conceivable or possible you could have these particular brain processes without phenomenal consciousness coming along for the ride — or you could kind of wonder whether, you know, there’s a possibility that it doesn’t, and so on. So it seems to me that illusionism is not kind of trying to make a claim about what we’ll discover in the course of doing neuroscience — although there are still quite interesting and important questions about what we’ll discover, which no one knows, including illusionists.

Henry: Then it sounds like — so, just very quickly, I’d say that the danger here, then, is that there’s a kind of motte-and-bailey thing going on: where the illusionist starts out with a very specific and very strong set of claims, within a much broader set of broadly reductionist, physicalist positions, and ends up saying, well, look: as long as there’s no supernatural stuff, right — or no fundamentally different substance, no non-physical property — then illusionism just wins, right? We claim all of physicalism as our territory, right? Whereas I think there are viable physicalist views, that we are just not in a position to evaluate yet, that are not illusionist in the strict sense of the term. And we just don’t know which of these views yet are correct.

Fin: Yeah — go ahead, then.

Dan: I think it’s worth wrapping things up with the following. So: you make a really good case in this paper, Fin, with Will, that, look — rapid AI progress, of the sort that is happening and is very likely to continue in the coming years and decades, is going to produce lots of artificial intelligences that are very sophisticated, have agency. Some of them, as I’ve said, are going to be kind of disembodied digital agents, in some sense. Some of them will be semi-autonomous, if not completely autonomous, robots. And whatever we think about these questions concerning the philosophy and science of consciousness, this is going to create a huge set of, like, ethical and political questions. How should we interact with these systems? How should we treat them? What kinds of societies do we want to build when we’re living in the context of non-human intelligences that are as cognitively sophisticated — maybe much more cognitively sophisticated — than we are?

So this is, like — to put it mildly — a grand challenge. And also one that gets basically no attention among people who are currently academics and researchers. Obviously there is a small community of devoted researchers who are thinking about this in a really serious way. But, like, relative to the scale of these challenges, it seems like it’s dramatically under-researched. So my view is, like: all of those challenges are real, and, in a sense, we can start thinking about those challenges — both intellectually and in terms of policies — without settling these really difficult questions about the nature of consciousness. So that’s how I would sort of want to wrap up that conversation. And also the message I would want to send to people who are listening to this or watching this: that this is a hugely important set of questions that is currently massively under-researched. So — do you two agree with that? Do you have anything to add?

Fin: I can try pitching in. I, unsurprisingly, do agree with that. One way of putting this is to say: settling questions on AI consciousness, and the metaphysics of consciousness — that strikes me as neither necessary nor sufficient to figure out, you know, how the hell to kind of arrange, you know, our institutions and society at large around, you know, this potential influx of digital minds, right? So: it’s not sufficient, because — let’s just say we have answers on, you know, which AI systems are conscious in some sense. Well, there’s another kind of, you know, kind of being where we have pretty confident answers — which is humans, right? And it turns out we still face questions about how to, you know, live together with each other, and associate with each other, and what kinds of, you know, legal, political institutions to build. And so just figuring out, you know, which systems are conscious does not tell us how to integrate with them. Should we ascribe them rights, or obligations, and so on? Those are just, as far as I can tell, largely independent — and extremely important, tricky, interesting — questions to answer.

And then I’d say it’s also not necessary to answer these questions — at least plausibly not totally necessary — in the sense that I can imagine, as we kind of approach a world where we do just, you know, live with some of these AI systems — we actually get to interact with them — it’s kind of plausible to me that some of these questions around consciousness will start to feel a little less kind of relevant, and will sort of kind of go from a kind of thought-experiment land into: I just, like, have these attitudes by default to these systems. In the same way where, if we watch a, you know, sci-fi movie where there’s an alien race, and they’re just very sympathetic — they seem very understanding and wise and empathetic — we just don’t tend to find ourselves speculating about whether they’re truly conscious or not. And we could imagine some of that happening as well. So that would be my kind of... my claim — which is not to say that these questions are not important, around consciousness. It just seems to me that there are so many other questions as well that we should get right.

Henry: So I agree with half of that. I certainly agree it wouldn’t be sufficient — insofar as I think our moral landscape is complex and pluralist, and we can acquire obligations that are not grounded purely in consciousness. You know, I think you might think we have obligations based on things like keeping promises and so forth, as well as, you know, maybe obligations grounded in relations to communities and so forth. And I expect, as sort of worlds of agents emerge — of ubiquitous AI agents that we’re forming close relationships with — yeah, there are gonna be a whole lot of moral questions that, you know, aren’t immediately settled by figuring out which ones are conscious and which ones aren’t.

That said, I do think it’s necessary, at least for a full moral theory, to have a sense of which beings are conscious — or — even if the answer to that is, like, just demonstrated conclusively that consciousness is the wrong property, right? That would be one way, right? But until we get an answer to that question, I think there’s gonna be this sort of yawning gap at the heart of our moral theories. Because pre-theoretically — and, frankly, theoretically, for maybe the plurality, if not the majority, of sort of ethicists and philosophers of mind — consciousness is a very, very, very, very special moral property. And to your example, Fin — I think, again, I sort of agree with you, right, about sci-fi and other shows. I think the reason that, you know, we don’t hesitate to ascribe moral status to Data, or, you know, the Klingons, or R2-D2 and so forth, is because we ascribe them interiority — we ascribe them an interior mental life, right? And I think that’s gonna happen: I absolutely expect that we’ll start to see widespread attributions of consciousness to AI systems by the general public. But that’s a different story from: we’ll just stop caring about consciousness. I think, instead, we will be convinced by the kind of thick relational properties that form between humans and AI systems — you know, that’ll drive our consciousness ascriptions.

Fin: Yeah — do you know what, I think that you’re right, in the sense that I think I do actually agree that in some important ways it is necessary to get clearer on metaphysical questions around consciousness. So maybe I should, yeah, reduce my claim down to just this point: that, very plausibly, once we enter this world where we get to interact with sophisticated AIs, it’ll feel potentially just quite natural and automatic how we should kind of relate to them and empathise with them. And we won’t be constantly asking ourselves, you know, which kind of metaphysics of consciousness is most plausible. I think it’ll, you know, potentially come quite organically. But we might be wrong. So—

Dan: Okay. With that, Fin — this has been great. Thanks for coming on.

Fin: Thank you both so much. Great questions.

Henry: Thanks so much, Fin. It’s been great.

Unless you’re a member of the billionaire class, you probably can’t afford to buy it at the price Oxford University Press have set (although this discount code might help a little: AUFLY30). However, some of you will be able to access it online or ask your university or local library to order it. And even if you can’t do any of these things, you still might be interested in hearing about its main ideas and arguments.

So, this post provides a summary, which is hopefully at least somewhat accessible. Although the book itself is very much an academic one, it is also highly interdisciplinary and written to be accessible to researchers across a wide range of fields, including philosophy, evolution, anthropology, psychology, neuroscience, psychiatry, and social science.

What the Book Argues

When people hear the title, they often assume it is a book focused solely on clinical delusions of the sort associated with conditions such as schizophrenia, bipolar disorder, dementia, and so on. An individual diagnosed as holding delusions in this sense might believe that their spouse has been replaced with an imposter, that they are the reincarnation of Napoleon, or that their neighbours are spying on them through their television.

A classic question in the psychological and medical sciences is where these strange beliefs come from. How can individuals become convinced of such ideas? And why would they hold onto them even when they are presented with so much apparent counter-evidence?

Although our book addresses these questions, they are only part of its concern.

It is also focused on collective delusions—what Charles Mackay famously called “extraordinary popular delusions and the madness of crowds”. Think, for example, of witch hunts, the Satanic panic, bizarre conspiracy theories like QAnon, or the strange supernatural myths of sects and cults.

Those swept up in these belief systems needn’t be mentally ill, and yet the beliefs often seem just as irrational as anything one might encounter in a clinic. What drives such outbreaks of collective madness? Why do communities of healthy, seemingly rational humans so often “lose their minds”?

This question invites a deeper conceptual puzzle: What does it even mean to classify a belief or person as delusional? Today, many psychiatrists and philosophers insist that the only real delusions are clinical ones. Are they right? Or might there be an underlying logic that explains why ordinary people are drawn to similar terms and ideas—mad, crazy, insane, delusional—whether the strange belief belongs to an individual or a collective?

Given the title, you can probably guess that our answers to these questions have something to do with human sociality. To understand delusions—why they emerge, what forms they take, and what the concept of delusion is even for—we must start not from a focus on what goes wrong inside individual brains but from the fact that we are a deeply social species.

Of course, merely saying “humans are social” is a platitude. The aim of our book, then, is to clarify the nature of this sociality and explain why it matters here, which we do by integrating a large body of research from evolutionary theory, anthropology, cultural evolution, psychology, social science, and epistemology.

Homo sapiens, we argue, is an ultra-social creature utterly dependent on complex systems of cooperation and culture to know and navigate our world. This dependence warps our psychology at almost every level and provides the background arena for our enduring conflicts and competitions.

This picture informs our account not just of delusion but also of human knowledge. Outside of narrow domains, knowledge of our reality is not the default state of human minds, even when they are healthy or functioning well. It is, instead, a fragile achievement of complex epistemic cooperation—of a vast collaborative enterprise of creating, communicating, and evaluating information—that rests on precarious networks of trust, testimony, reputation, and a division of intellectual labour.

This social-epistemic machinery, we argue, is our species’ epistemic superpower, but it is also the source of our vulnerability to becoming delusional.

Subscribe now

Keynes vs Nietzsche

We illustrate this dual character of human sociality with two quotes:

The first comes from John Maynard Keynes:

“It is astonishing what foolish things one can temporarily believe if one thinks too long alone.”

The second comes from Friedrich Nietzsche:

“Madness is something rare in individuals—but in groups, parties, peoples, and ages, it is the rule.”

Keynes’ quote draws attention to our epistemic dependence on others, which manifests in at least two ways. On our own, we can discover and understand very little of the world. Outside of the narrow domain of our first-hand experience, we rely entirely on others to learn about reality. This, of course, is a central lesson of research on social epistemology and cultural evolution.

Less obviously but just as importantly, solitary reasoning is treacherous. Although we have powerful capacities for reasoning, Hugo Mercier and Dan Sperber have documented what typically happens when we reason alone: freed from pushback and correction from others, we typically just manufacture additional confirmations and rationalisations of what we already believe.

However, this deep reliance on others also makes us highly vulnerable to deception and manipulation. To navigate this dependence, we have evolved capacities of “epistemic vigilance” through which we evaluate the plausibility of what we are told, the trustworthiness of testifiers, and the quality of their arguments. The problem is that this whole arrangement is fragile: if it breaks down—for example, through isolation, mistrust, and social disconnection—bizarre or “foolish” beliefs can come easily.

This lies at the core of our story about clinical delusions. We emphatically do not deny that things often go “wrong” inside the brains of delusional individuals in such cases: for example, perceptual abnormalities, or disturbances in how prediction errors are processed. But we argue that these disturbances become catastrophic precisely when a person becomes cut off from the corrective testimony and feedback of others, at which point solitary reasoning and sense-making start elaborating and rationalising the resulting beliefs.

Next, turn to Nietzsche’s quote, which highlights the other face of human sociality. Although the machinery of epistemic cooperation lies behind our capacity to know and navigate the world, and fuels our most spectacular epistemic achievements, it also enables perfectly rational, healthy individuals to co-create spectacular forms of madness and irrationality. Our book also aims to tell that story.

The Book’s Three Parts

The book develops these ideas across three parts.

Part 1 first develops the core scientific framework underlying everything else in the book: a picture of human beings as deeply cooperative, cultural, and competitive apes. We can’t survive or succeed alone; we acquire most of what we know from others; and we’re locked in conflicts of interest and competition for status, which makes us crave social approval and esteem and leaves us highly vigilant for deception and manipulation. I’m proud of this framework: if you want a clear, scientifically informed account of our evolved human nature and sociality, I think it is a good place to start.

The book then draws on this framework to understand the concept of delusion. Traditional accounts of the concept often look for neat psychiatric or epistemic criteria to tell us what delusions really are. Against this, we argue that treating someone as delusional (even when people don’t use that specific word) is less a diagnosis than an expression of bafflement at the relevant beliefs and a signal to others that the person in question is a “rational lost cause”—someone unreachable, on the relevant topic, by rational argument and persuasion.

Of course, once rational persuasion is off the table, we might be drawn to non-rational attempts to change the person’s mind, which is the approach adopted in modern medical psychiatry. However, illness or pathology is not essential to the concept in our view. We can and do express the same bafflement, and recommend the same kind of rational disengagement, when confronted with strange non-pathological beliefs—most obviously, in the collective delusions already mentioned. And in other times and cultures, people have sought spiritual interventions (e.g., demonic exorcism) for such beliefs, not medical ones.

Part 2 asks where collective delusions come from. Against theories that attempt to answer this question by looking at in-the-head cognitive biases, wishful thinking, or simple social learning biases, we argue that the most striking examples of collective delusions must be understood in terms of the strategic, social goals they serve. This includes goals such as advocacy or “propaganda” (e.g., witchcraft accusations and bizarre conspiracy theories that demonise target victims in ways that justify their exploitation) and social signalling (e.g., doctrines of cults and sects that signal group membership and allegiance).

Importantly, none of this requires conscious dishonesty. Sincerity and strategy, we argue, are not opposites: people often internalise those beliefs that help them persuade others, support allies, demonise enemies, or display their social allegiances.

When these motives are distributed across whole communities, they invert the machinery of epistemic cooperation, redirecting it away from the goals of learning and inquiry to the task of promoting and protecting shared fictions. People co-create “epistemic reward systems”—systems of social incentives, status competition, and a division of intellectual labour—that support forms of collective irrationality no individual could achieve alone.

Part 3 then turns to the clinic. As noted, our framework for understanding these beliefs does not deny that individual disturbances or psychological malfunctions often play a significant role. For example, they typically generate strange perceptual and affective experiences that people struggle to explain, and they can distort how the brain processes information.

Nevertheless, we argue that an exclusive focus on what goes wrong inside individual brains overlooks three of these beliefs’ most striking social features:

• social adversity—abuse, discrimination, marginalisation, humiliation—significantly increases people’s risk of developing them;

• they focus overwhelmingly on social themes such as surveillance, conspiracy, control, and status;

• they are peculiarly resistant to social evidence—to the testimony of family, friends, and doctors.

This latter characteristic is especially surprising in a species so reliant on social learning. If strange experiences lie at the root of clinical delusions, why do delusional individuals prioritise the misleading evidence of those experiences over the consistent corrections they receive from seemingly trustworthy people?

We address this and other puzzles by appealing to the complexities and vulnerabilities of human ultra-sociality.

For example, the social world we must navigate is uniquely complex, opaque, adversarial, and high-stakes. Unlike physical objects, rational agents can conceal their intentions, coordinate with others in secret, manipulate our evidence, and execute long-term strategies. Hence, not only do our minds instinctively gravitate towards social explanations of unusual events, but these explanations are uniquely challenging to disconfirm. What would disprove the worry that you’re the victim of a carefully concealed long-term plot?

We also explore the many reasons why people suffering from psychological distress, disturbances, and social exclusion can become cut off from the corrective social feedback on which we normally depend to regulate our beliefs. In such cases, the epistemic vigilance that ordinarily protects us from manipulation and deception ends up protecting beliefs from disconfirmation, and the biased, lawyerly character of reasoning that usually helps us persuade gets to work in elaborating and rationalising those beliefs.

The Blind Leading the Blind

This is a brutally compressed summary of a 120,000-word book, but hopefully it gives a fair overview of what the book is about.

It also helps explain the book’s cover: Bruegel’s The Blind Leading the Blind.

For us, the painting is not a metaphor for mental illness. It captures a deep truth about the human condition. On our own, there is an important sense in which none of us can see. To learn about the world, we must rely on others, who are in precisely the same situation.

The painting also highlights a fundamental explanatory inversion at the heart of our project.

The puzzle of delusions is sometimes framed as asking why people “lose contact with reality”. For us, however, seeing things clearly—being “in contact” with reality—isn’t the default state of human minds. It’s a complex, fragile, and highly fallible achievement.

The fundamental puzzle, then, is not why some people believe false things, but how any of us, some of the time, manage not to.

We hope that this inversion both demystifies and destigmatises delusions.

Find the book here

The central thesis of Thomas Sowell’s ‘A Conflict of Visions’ is that an enduring conflict between two competing visions of human nature and society drives much of the high-level disagreement we observe in politics.

According to the “constrained” (or “tragic”) vision, humans are deeply and permanently morally and intellectually limited. We are self-interested, biased, and ignorant of most things outside the scope of our own concerns and experiences. So, we must build societies around these constraints of human nature, either by making them too costly to act on or by harnessing them for beneficial ends. This, argues Sowell, is why proponents of the constrained vision prefer free, competitive markets to central planning of the economy: not only do market incentives channel the self-interest of butchers, brewers, and bakers into delicious dinners, but market prices coordinate dispersed knowledge that no humans, no matter how intelligent, could ever possess in full.

Those who embrace this tragic vision are not surprised by poverty, conflict, disorder, or ignorance. Given our moral and intellectual limitations, these are the default conditions of humanity. Instead, the deep challenge for social theory is to explain how fallen creatures such as us could ever escape these conditions and achieve order, peace, and prosperity.

The “unconstrained” (or “utopian”) vision takes a more optimistic view of human potential. Its advocates do not deny that humans can be—and, in the present state of political development, often are—selfish, immoral, and ignorant. However, they ascribe these outcomes not to permanent limitations of human nature but to contingent institutions, specific false beliefs, or a curable lack of moral and intellectual enlightenment.

Instead of resigning ourselves to human self-regard and ignorance, the task of politics is therefore to apply our rational and moral faculties to the design of societies that cultivate and harness our potential. Hence, those with this vision are liable to dream of abolishing not just the profit motive and other competitive social institutions but also, one day, the police and prisons. To the extent that self-interest, and hence conflicts of interest, are products of specific social arrangements or an immature stage of intellectual and moral development, crime and violence would simply disappear in a world of true enlightenment, abundance, and equality.

Visions and Politics

Sowell argues that people’s commitment to these visions explains something that should otherwise be puzzling: why political disagreements about seemingly unrelated issues (economics, crime, law, geopolitics, tradition) so often cluster around two broad camps. For example, why do people’s beliefs about free markets predict their beliefs about law and order, traditional cultural norms surrounding gender, and warfare? Sowell argues that broadly “conservative” or “right-wing” positions on such issues trace back to an underlying constrained vision of human nature and social causation, whereas broadly “liberal” (in the American sense), “progressive”, or “left-wing” positions trace back to the unconstrained vision.

Admittedly, Sowell is quite cautious in presenting this claim. He acknowledges that visions are ideal types, that the visions exist on a continuum, that “hybrid” visions exist, and that real-world political coalitions are shaped by many non-intellectual factors. Nevertheless, he clearly sees it as a significant virtue of his analysis that it can explain the clustering of political beliefs around broadly conservative/right-wing and liberal/left-wing viewpoints.

This part of the analysis is a failure. In general, real-world “ideologies” are less like applications of deep assumptions than ad hoc collections of principles, beliefs, and rationalisations that suit the interests of specific alliances in specific political contexts at specific times.

To take only one example, Sowell takes for granted that the right/left divide bundles together economic and socio-cultural concerns in a distinctive way: the right, or “conservatives”, prefer free markets and cultural conservatism, whereas the left favours top-down management of the economy and cultural liberalism. He then tells a story in which this package reflects the coherent application of deeper visions about human nature and social causation.

Writing in the 1980s, this all probably seemed self-evident. The problem is that this bundling is, in fact, very historically and culturally contingent. Globally, the more common pattern seems to be for economic liberalism (i.e., free markets) to correlate with social liberalism, with cultural conservatives often favouring more state intervention in the economy, not less. Moreover, one can easily tell a story in which combining economic with social liberalism in this way makes more rational sense than the bundling Sowell takes for granted as the coherent expression of a “constrained” vision of human nature and society.

What seems to have happened is that the meaning of “conservatism” throughout the Western world after the Second World War had more to do with a specific alliance between business elites and cultural conservatives than with any underlying vision of human nature and society. It is noteworthy, for example, that the meaning of “conservatism” even in the US today is very different from the package of views Sowell took for granted in the 1980s of Reagan and Thatcher, plausibly because of shifts in the underlying alliances that increasingly shape Western politics.

However, in some ways this analysis is too favourable to Sowell. It suggests that the constrained and unconstrained visions as he describes them are rationally coherent, and that he only went awry in trying to map them onto real-world political belief systems. In reality, I think the problem with his analysis is deeper: the visions themselves combine ideas that don’t bear any rational connection to each other. Specifically, his “constrained vision” of human nature pairs two very different kinds of constraints: moral constraints on the one hand, and what philosophers would pretentiously but accurately call “epistemic” constraints—that is, constraints on our ability to know and reason about the world—on the other.

There is no obvious reason to think that these two constraints must go together. In fact, whereas I think we are deeply morally limited—there is a self-interested, nepotistic, and competitive core of human nature that, short of radical bio-hacking in the future, we will never escape—our collective capacities for reason and knowledge are, in an important sense, open-ended. We are grubby, self-regarding, devious little primates, but we are not consigned to permanent ignorance. And this fact, I think, has important implications for how we should think about politics and social theory.

Our Deep Moral Limitations

It’s always been obvious that if human nature involves permanent moral or altruistic limitations, this will have enormous political implications.

For example, although Plato’s Republic—in many ways, the foundational text of Western political philosophy—is primarily a book about the nature of justice and a just society, much of it also concerns questions of human psychology.

In Book 1, the character of Thrasymachus advances a classic cynical perspective on such questions. If you strip away all the lofty rhetoric surrounding justice, he argues, it boils down to a question of strength: people’s judgements about justice typically reflect their self-interest, so what ends up being called “justice” in a given society is nothing but the advantage of the stronger.

(Plato—well, his mouthpiece in the character of Socrates—rejects the idea, and The Republic is often taught as if Thrasymachus’ views are obviously wrong, but it’s noteworthy that Plato was writing in an era in which perceptions of “justice” coincidentally favoured an extremely hierarchical aristocratic, patriarchal, and slave-owning society. And it’s amusing that Plato’s own model of a just society just happens to hand immense power to… philosophers.)

In Book 2, the character Glaucon presents a more nuanced cynical challenge, illustrating it with the famous story of the Ring of Gyges, which makes its wearer invisible. If a seemingly noble and just man wore such a ring, Glaucon argues, the loss of any reputational motives would cause him to behave like an unjust man. And that’s because justice and morality are fundamentally rooted in self-interest and reputation management. Take away the incentives that sustain this structure and noble, altruistic motives will dissolve along with them.

That is one of the earliest and clearest statements of the constrained vision of human motivation. And in this form, at least, it is sufficiently bleak and extreme—Glaucon is clear that he doesn’t want to believe it and offers it up precisely to learn why it is mistaken—that a vast amount of political philosophy from Plato onwards has aimed to refute it.

In reality, it is possible to endorse a constrained vision of human altruism without thinking that humans are exclusively selfish. For example, Sowell’s own preferred exemplar of the constrained vision comes from Adam Smith. It’s a good choice, not just because of Smith’s famous arguments about how self-interest can, under the right conditions, be channelled into economic activities that benefit everyone, but also because his ‘The Theory of Moral Sentiments’—the hipster’s favourite Smith book, published seventeen years before ‘The Wealth of Nations’—contains an extremely rich and nuanced moral psychology.

Smith opens this book by observing that “how selfish soever man may be supposed, there are evidently some principles in his nature, which interest him in the fortune of others.” Although his subsequent account of our moral psychology contains too much richness and complexity to be summarised here, the core idea is that our basic capacity for “sympathy” (very roughly, empathy in today’s terminology) and our craving for social approval lead us to internalise an “impartial spectator”, a kind of inner judge whose verdicts we consult even when nobody is watching.

One way of understanding this is that whereas ‘The Wealth of Nations’ focuses on the formal economy of goods and services, ‘The Theory of Moral Sentiments’ focuses on how morality emerges from the informal prestige economy of social life and its roots in basic processes of sympathy, social evaluation, and social comparison. (Smith’s analysis prefigures Christopher Boehm’s more cynical evolutionary account of the human conscience as “a social mirror. By continually glancing at it, we can keep track of shameful pitfalls that threaten our reputational status or proudly and virtuously chart our personal progress as group members in good standing”).

For Smith, then, humans are not purely self-interested sociopaths. But we are substantially and unavoidably self-interested nevertheless, and our prosocial instincts, while real, are wrapped up with social comparison and reputation management in complex ways, and often warped by partiality, hypocrisy, and self-deception.

This is ultimately why he favours political and economic systems that economise on virtue, especially when such systems connect strangers not bound together by strong bonds of family and friendship, or the immediate reputational pressures of a face-to-face, small-scale community. It is not because we are incapable of virtue, but because human virtue has unavoidable limits that impose non-negotiable constraints on what kinds of societies are possible.

Modern evolutionary and empirical research has largely vindicated this picture, at least in broad outline. The utopian hope that self-regard, social competition, and moral limitations are simply products of bad institutions or a failure of proper “education” is impossible to believe today, not just because such an unconstrained moral psychology literally could not evolve via natural selection, but also because of what we have learned by researching human beings, both through the study of history and different cultures and through more systematic scientific research.

Humans have genuine moral and prosocial motivations, some of which are plausibly innate, but these motives are highly selective and conditional; they are frequently distorted by hypocrisy, self-serving bias, and motivated reasoning; and they are often outweighed by self-interest.

Interestingly, if Smith erred, it was by exaggerating the impartiality of our inner spectator (it is, at best, impartialish) and by downplaying just how much status competition and reputation management shape the nature and intensity of our moral concerns.

For example, the very ideal of an impartial spectator—closely connected to what social scientists call “impersonal prosociality”—is itself culturally specific, plausibly emerging from reputational incentives in individualistic, fluid social ecologies in which people must be able to build cooperative relationships with strangers. (This includes the commercial society Smith inhabited and so astutely analysed). Throughout most of human history, people’s social world was dominated by kinship and other enduring personal relationships, which incentivised the development of moral instincts—and, hence, inner spectators—that are extremely partial, assigning much greater weight to the biased judgements and interests of kith and kin than to those of outsiders and strangers.

Sowell is correct that this constrained vision of human nature has profound political implications. We (or at least most of us) are not purely strategic and self-interested, and complex cooperation would be impossible if we were. However, we will also never be saints, and functional societies must find ways to productively manage deep-rooted, ineradicable instincts towards self-regard, nepotism, and status competition. As Machiavelli put it, “He who neglects what is done for what ought to be done, sooner effects his ruin than his preservation.”

Subscribe now

A Tragic View of Knowledge?

But what does any of this have to do with knowledge? For Sowell, the constrained vision combines this tragic view of our moral limitations with a tragic view of our epistemic limitations.

According to this tragic vision of our epistemic limitations, Sowell argues, we are permanently limited in what we can know as individuals. We do not know enough on our own even to govern our own lives, let alone a whole society. Civilisations therefore depend on mechanisms—traditions, habits, laws, norms, prices—that transmit and coordinate the experiences of millions, including those of previous generations, winnowed by trial and error over long periods of time.

Moreover, this socially accumulated knowledge is often implicit and difficult, if not impossible, to articulate and render explicit. It involves habits, practical skills, emotional dispositions, and ways of doing things that people struggle to explain. It is also embedded in collective customs, rituals, traditions, and institutions that often perform valuable functions that nobody consciously designed or even fully understands.

For example, people are often heavily invested in upholding seemingly arbitrary manners and participating in strange, seemingly irrational communal rituals without appreciating the important roles these practices play in sustaining mutual trust and cooperation. Similarly, the prices of goods encode systemic information about their scarcity, demand, and alternative uses that no buyer or seller needs to know, or could know, in full. As Hayek argued, the knowledge is dispersed across society and embodied in society-wide patterns of behaviour, not located inside anyone’s head.

On this view, then, the socially useful knowledge upon which societies depend mostly involves inarticulate social experience—“wisdom without reflection”, in Burke’s phrase—that is widely dispersed and impossible to concentrate. Hence, this vision is typically highly sceptical of intellectuals, central planners, and any system of values that restricts our assessment of ideas and practices to explicitly articulated reasons.

In contrast, the unconstrained vision adopts a very different epistemology. It argues that socially useful knowledge paradigmatically consists of explicit information—the kind one could read in a book—and defensible explicit reasoning. If ideas, practices, norms, or institutions were not designed according to rational principles, they should be viewed with suspicion; to the extent that they cannot be rationally defended, they should be rejected. So, advocates of this vision have a much more positive view of experts and intellectuals and encourage deference towards enlightened decision-makers, top-down planning, and deliberate institution design.

An Assessment

Although Sowell is right that this conflict of epistemological visions accurately captures the views of certain intellectuals—thinkers like Hayek and Burke obviously lie on the constrained side, whereas many generations of Enlightenment “rationalist” philosophers fall on the unconstrained side—there is a considerable amount of confusion in his analysis.

To start with, the constrained vision bundles together multiple different claims that don’t bear any obvious rational relationship to each other: for example, (1) that individuals are ignorant of most things; (2) that valuable knowledge is often socially distributed, tacit, and accumulated across generations; (3) that decentralised systems can sometimes coordinate and apply knowledge better than central authorities; and (4) that articulated reasoning and deliberate institution design should be viewed with general suspicion.

It is not at all obvious why one should treat these as a package deal. For example, the first two claims seem clearly correct, the third is complicated and plausibly depends on the details of specific cases, and none of them establishes the fourth.

Moreover, it is very unclear what any of them have to do with the constrained vision of moral motivation. There doesn’t seem to be anything incoherent in thinking that humans are deeply morally limited whilst also being highly sceptical of tradition, custom, and free markets, optimistic about articulated rationality, or enthusiastic about deliberate institution design. And in fact, some philosophers embrace a vision of moral constraints without embracing the vision of epistemic constraints that Sowell describes (e.g., Hobbes), whereas others embrace a broadly Hayekian vision of knowledge whilst embracing a much more idealistic view of human moral motivation (e.g., anarchists such as James C. Scott).

A defender of Sowell might concede that such dissociations are possible in principle but nevertheless argue that the two kinds of constraints go together as follows: if we are permanently self-serving, competitive, and morally limited, we should expect these motivational properties to corrupt our reason. For example, perhaps self-interested, status-seeking, and tribal primates are more likely to use reason not as impartial judges but as lawyers, advocates, propagandists, and public-relations agents.

In fact, I think this expectation is correct. However, it doesn’t vindicate Sowell’s constrained vision of knowledge.

First, to the extent that we are self-regarding and morally limited, this will distort not only our explicit reasoning but also our traditions, customs, rituals, social practices, and culturally evolved institutions. Sowell is surely right—as was Burke—that such things often perform latent functions that people cannot consciously articulate. But if one endorses a morally constrained vision, one should expect many of those functions to have more to do with domination, extraction, and oppression than with mutually beneficial cooperation.

For example, it is quite difficult to explicitly articulate or understand the rationale for the centuries-long practice of foot binding in China, but its latent “function” was hardly conducive to advancing the common good. Many norms, rituals, and cultural practices are like that.

Second, if we should be sceptical of articulated reasoning on motivational grounds, that scepticism must also extend to the reasoning of those intellectuals who advocate the constrained vision of knowledge. Figures such as Burke, Hayek, Oakeshott, and Sowell himself do not merely blindly defer to tradition; they articulate reasons for their deference. There is no obvious reason why their arguments should be exempt from the distortions of self-interest and bias that supposedly afflict everyone else.

Third, one can happily accept that humans are morally limited and often self-regarding in their reasoning and conclude that the solution lies in careful norms and incentive structures that channel ambition and bias into collectively beneficial epistemic outcomes.

There is a clear analogy with the economic domain here: just as markets do not presuppose benevolent butchers, brewers, and bakers, well-designed epistemic institutions do not and should not presuppose disinterested truth-seekers. In fact, this is precisely how science works: it is a status game, a complex prestige economy in which esteem and recognition are awarded for making discoveries, exposing errors, and solving challenging intellectual problems. By combining the self-belief, competitiveness, and vanity of scientists with institutionalised norms of scrutiny, criticism, and replication, science can advance the collective frontier of knowledge without relying on any individual scientist to aim specifically at that outcome.

Of course, this process often goes wrong in many ways. Scientists can also win status by publishing flashy nonsense, manipulating data, taking credit for other people’s ideas, and following fashions. The replication crisis and other systemic challenges facing modern science illustrate these problems. Moreover, elsewhere Sowell himself offers an astute analysis of how bad incentives ensure that intellectuals are often overconfident and biased in their theories and recommendations, responding more to internal criteria of fads, fashions, and moral grandstanding than to external criteria of objective accuracy or predictive success. However, this analysis simply concedes that the ultimate issue here is incentives, not permanent limitations: with superior norms, rewards, and institutional procedures, we could produce superior epistemic outcomes.

In fact, the scientific response to the replication crisis illustrates this very lesson. After identifying how incentives in science encouraged unreliable methods, reforming scientists advocated for (and continue to advocate for) innovations such as preregistration, registered reports, open data, better statistical standards, and greater rewards for replication efforts, all of which improve the alignment between individual incentives and knowledge creation.

Similar lessons could be applied more broadly to intellectuals, including public intellectuals. For example, Philip Tetlock has argued over many years that a deep problem with our intellectual class concerns their reputational incentives: they are almost never held accountable for making bad predictions, not least because they are rarely expected to be precise in stating them. Instead, they are rewarded for pontificating in bold, overconfident ways that audiences find entertaining or politically useful. In principle, we could enforce norms and design technologies that track the accuracy of intellectuals’ claims and thereby change these incentives. No alterations to human nature would be necessary.

These reflections connect to the fourth and final issue.

Like Burke and Hayek before him (and Henrich since), Sowell is clearly correct in drawing attention to the limits of human knowledge and explicit reasoning, the benefits of social systems that aggregate dispersed knowledge, and the importance of inarticulate social experience, all of which are often overlooked by intellectuals. He is also clearly correct that some policies and political programmes presuppose absurd claims to knowledge. Economies really cannot be centrally planned from the top down, and socialists and technocrats who think otherwise are making a mistake.

However, no coherent epistemology, “constrained” or otherwise, holds that humans can know anything and everything, or that explicit reasoning delivers knowledge relevant to every goal. Ultimately, what we want in politics is, therefore, meta-knowledge: knowledge of what we can and cannot know, and in which circumstances. When is knowledge too dispersed to be centralised? When do prices reveal socially useful information, and when are they distorted by market power or externalities? When do traditions embody latent “wisdom” or perform valuable hidden functions, and when do they merely advance the relative interests of specific elites, races, classes, castes, or other groups?

Of course, these questions are extremely difficult to answer. But equally obviously, they are questions that we will only ever answer with explicit inquiry, data collection, scientific methods, and rational argument. Hence, even Hayek—and, elsewhere, Sowell himself—advance explicit rational arguments for the limits of articulated knowledge and rationality in specific cases. As others have noted, it is precisely this ability of human reason to reflect on its own conditions and limitations that makes it so powerful, and so indispensable in the domain of politics and social theory.

Although our meta-knowledge will always be partial and fallible, it need not be permanently fixed. Creating wealth is also extremely challenging, and yet humans have achieved spectacular material abundance through accumulated innovations, specialisation, exchange, and institutions refined over many years. Explicit, rational knowledge can be created in a similar way. Our permanent moral and individual cognitive limitations need not impose a fixed ceiling on such knowledge, any more than they must consign us to permanent poverty.

There is a broader point here. Those sympathetic to free markets and cultural conservatism, such as Sowell, often present their viewpoint as a manifestation of epistemic humility, in contrast to the epistemic hubris of rationalist progressives. This is fundamentally confused. The claim that culturally evolved practices are more beneficial than explicitly designed ones, or that market economies are superior to top-down central planning, might well be correct, or at least correct in specific circumstances. But such claims are just as epistemically ambitious as their denials. They involve substantive, contestable theses about how complex social systems work, and they can only be established through evidence and explicit reasoning.

Conclusion

One can—and, I think, should—embrace a tragic vision of human moral motivation without embracing a tragic view of reason or knowledge.

We will never educate or reform our way into a species of impartial altruists unconstrained by the grubby forces of self-interest, status, personal attachments, or self-deception. But these limitations alone do not doom us to ignorance. With the right norms, incentives, and institutions, we can channel the flaws of human nature into the open-ended creation of knowledge and understanding, including wisdom about the limits of our knowledge in specific circumstances. In fact, it is only through this sustained intellectual progress that we have learned that a tragic view of our moral limitations, and hence the importance of careful institution design, is not just a subjective “vision” of certain bygone intellectuals but a genuine discovery of modern science.

This is a more coherent “tragic” vision than the one described by Sowell. Of course, it doesn’t map very well onto any of the major real-world ideologies we encounter today on either the left or right. But fortunately, this misalignment is just what the vision predicts: real-world politics is constrained more by the grimy practicalities of power, alliance formation, and propaganda than by coherent philosophies.

Its central claim is that Europe has made three big mistakes. It has underestimated the speed of AI progress, underestimated the scale of its effects, and overestimated how easily it could catch up later. Unless this changes, the authors argue, Europe may soon find that the AI models powering its economy are controlled by a small number of American companies, run on American infrastructure, and ultimately subject to decisions made in Washington.

The scenario was published on 11 June. The next day, the US government imposed export controls on Anthropic’s two most powerful models. The order formally prohibited access by foreign nationals, but because there was no practical way to enforce that restriction, Anthropic was forced to suspend access for everyone. The controls were later lifted, but the episode offered a well-timed demonstration of some of Europe 2031’s central concerns.

Our guest is Judith Dada, one of the scenario’s authors.

Judith is co-CEO of Langdock, an enterprise AI company, and a senior partner at Visionaries. She has spent nearly a decade investing in European technology companies and advises the German government on AI transformation.

Henry and I ask Judith how plausible the scenario’s assumptions are, whether narrative scenarios can actually tell us anything about reality (and what would falsify them), and why Europe should prioritise data centres rather than simply trying to build or fund its own frontier AI company.

Judith is more uncertain about the future than a short summary of Europe 2031 might suggest. She concedes that nobody knows how quickly technical progress will diffuse through slow-moving organisations and institutions. But she is convinced that Europe must prepare for a world in which advanced AI is extremely valuable, and in which access to such AI becomes a major source of economic and geopolitical power.

We discuss

• Why the authors chose to present Europe 2031 as a fictional scenario rather than a conventional policy report

• What a narrative scenario can actually establish if it is not intended as a prediction

• Whether Europe really is underestimating the speed and significance of AI progress

• The difference between rapid improvements in AI capabilities and the much slower process of adoption by companies, governments, and institutions

• Whether powerful AI could make badly designed institutions (or simply institutions adapted to a pre-AI world) less effective rather than more productive

• Why Judith thinks Europe should aim to host roughly 15 per cent of the world’s AI compute

• Whether data centres on European soil would actually provide meaningful geopolitical leverage

• Why Europe cannot rely solely on the leverage provided by critical semiconductor-equipment companies such as ASML

• The case for alliances among European countries and other ‘middle powers’ such as Japan and South Korea

• Why Europe should not simply concentrate its resources on building a frontier AI company capable of competing with OpenAI and Anthropic

• Whether Europe’s energy prices, planning systems, and regulatory environment make a massive compute build-out challenging

• The unavoidable trade-offs between economic growth, technological sovereignty, environmental constraints, and local opposition to data centres

• Whether Europe’s emphasis on AI safety is compatible with a much more ambitious industrial strategy

• What intelligence will be worth if it becomes abundant

• Which human abilities, relationships, and experiences may become more valuable as intelligence becomes cheaper

• What a more hopeful European AI scenario might look like, and why it remains harder to describe than disaster

Links

• Europe 2031: the full scenario

• Executive summary

• The compute forecast behind the scenario

• About the authors and the project

• Judith’s Substack, Dadalogue

• AI 2027, which inspired the format

• AI as Normal Technology — Arvind Narayanan and Sayash Kapoor

• Anthropic’s statement on the US government’s temporary restriction of its most powerful models

• The UK AI Security Institute

Transcript

• Please note that this transcript is lightly AI-edited and may contain minor mistakes.

Introduction

Henry

We are delighted to be joined today by Judith Dada, one of the authors of Europe 2031. Judith is co-CEO at Langdock, one of Europe’s leading enterprise AI companies, and senior partner at Visionaries Club, a leading European venture capital fund. Previously, she was general partner at La Famiglia, the European early-stage venture capital fund that merged with General Catalyst in 2024. With nearly a decade of experience investing in AI, Judith has backed companies including Langdock, Parloa, Black Forest Labs, and Personio, where she also serves on the board of directors. Her career began at Facebook, where she oversaw Amazon Europe’s marketing strategy and Facebook’s VC Initiative. She advises the German government on AI strategy, writes regularly about AI and its impact on society, and is passionate about strengthening Europe’s leadership in artificial intelligence.

And today she joins us, as I noted, as one of the authors of Europe 2031, a fictional five-year scenario of Europe’s slide into AI irrelevance, which has gone incredibly viral for a 20,000-word policy document. I myself picked it up, started to skim it, and then was still sitting on a train platform an hour and a half later because I was so sucked in by the doc. And it’s now been read in the European Parliament, it’s been raised in UK–German talks and published, interestingly, with excellent timing, the day before we started to see some of the strongest transatlantic moves on AI regulation, with a Washington export order briefly taking Anthropic’s newest models offline for everyone, Americans included. So with that prestigious intro on the books, Judith, welcome to Conspicuous Cognition.

Judith

Thank you so much. I loved how upbeat that introduction was, given the heavy topics we’re going to discuss. It’s great to be here.

What is Europe 2031?

Henry

Fantastic. So for listeners who haven’t yet read Europe 2031, do you want to give us, in whatever format you prefer, an overview of what it’s about?

Judith

So Europe 2031 is what we call a narrative scenario. We didn’t want to write it as a report. We wanted to write it as a story, but we also wanted to make clear that while it is a story that focuses on the narrative, there is a scenario element to it that very much is grounded in deep research and a lot of discussion around, you know, making the piece logically consistent and something that describes a future that we could see unfolding going forward. We wrote Europe 2031 as a scenario that follows two characters. The first is Caroline; she’s a young policy worker in Brussels. The second is Christian, who is a German founder of a fast-growing AI company who’s based in Silicon Valley. And the story focuses on text messages that the two of them are exchanging and kind of their lives unfolding over the course of five or six years.

And they kind of keep, you know, this transatlantic relationship, friendship, in terms of, you know, kind of Christian not quite understanding how Europe can get all these things so wrong. And Caroline, you know, kind of from within, you know, kind of the European policy machine describing her frustrations, but also really her fight in terms of trying to alert people to the realities of AI and their implications. And so, in a nutshell, what Europe 2031 describes is a European continent that vastly underestimates the speed at which AI is unfolding. You know, Europe kind of thought we had decades when really we only had years. Secondly, it underestimates the magnitude of AI. We kind of thought of AI as just another technology, maybe the same way we thought about smartphones or the desktop computer.

When really the AI revolution, you know, as we describe it in the scenario, unfolds much more like an industrial revolution, where it’s kind of a wave that builds that at one point doesn’t leave any piece of ground untouched and you know, kind of not wet. It very much is transformative. We think about it much like we think about electricity, in terms of you know, kind of the underlying infrastructure that at some point silently powers everyone’s lives. And then the third, you know, kind of Europe consistently takes the wrong decisions because a lot of the people speaking the truth about AI progress are messengers that Europe is deeply skeptical of. You know, it kind of is, we call them power-hungry men who are based in Silicon Valley.

You know, it’s kind of a small elite of companies, you know, that in many ways over the last couple of years have also become entangled with the US administration, with a lot of the political tendencies that, you know, the European governments are deeply skeptical of and so they kind of discard everything that those messengers are saying as you know untrustworthy and you know kind of unnecessary hype without actually being able to separate that one can be critical of some of the political things that you know these actors are doing while at the same time very much taking seriously the technological actions and predictions that they make for the future.

And so you know Europe 2031 is a story of Europe underestimating AI and all those dimensions and that leads to us being economically sidelined, politically sidelined, and at the end of the day the European alliance as we know it today, fragmenting because every country is scrambling to, you know, kind of find its footing in this new reality in which the alliances that previously held no longer hold and in which every country is kind of trying to you know fight for the scraps that remain in terms of protecting its people and finding its role in this new geopolitical reality.

Dan

So Judith, before we get into the kind of details of the scenario and some of the assumptions, could you say a little bit about who the authors are?

Judith

Yeah. I like to refer to our author group as like the EU Commission if it worked well. So we’re kind of a pan-European group of authors. Some of us come from the kind of policy research side. Some of us come from the AI research side. And then you’ve got a wonky little Judith in there who’s like an investor and comes maybe, you know, from the capitalist side of things. And we all found each other through various connections. I would say kind of, you know, in the AI-pilled policy research circles in Europe. And then they found me through some of my unhinged retweets and tweets on X. They kind of were like, who is this woman and why is she like retweeting all this AI-pilled stuff? We don’t see a lot of people in Germany doing that. And so I had read AI 2027 when it came out.

And I was just moved by it. Like I couldn’t stop thinking about it. And back in the day I said, I want to write this for Europe. And I started doing so, but it was just me, myself, and I. So I didn’t quite get as far as I wanted to. And that’s when I kind of bumped into this group of kind of people who were coming more from the policy research side. And then we joined forces in basically writing the scenario. There was a lot of pre-work. I mean, part of the group had put out a report, much more kind of an actual report, an actual, thoroughly researched piece, more about kind of the implications of AI, but kind of in that more neutral, you know, kind of policy tone. And while it got some reception, some people read it and thought it was really interesting.

It didn’t quite reach the same awareness level as Europe 2031 did. And I think what we all shared was this passion for storytelling and just wanting to make this something different, something that really has a chance of kind of you know smashing through the Overton window of awareness and attention on you know how people talk about technology. And one thing that we loved about AI 2027 is that, you know, many of us loved it, but also had a ton of things that we disagree with, but it allowed you to kind of see the future in concrete terms; it gave the future a concrete shape. And you could say, I don’t like the little bubble over there, and I disagree with the little, you know, kind of edge that you’ve drawn there.

And so we just fiercely believe in having to make things concrete in order for people to start forming an angle towards that kind of concrete shape that you have put out in the world. And so we wrote this very much not wanting it to be a prediction. We very clearly said that we’re not superforecasters, we have much respect also for the work that Eli and Daniel are putting into their predictions. Like, I mean, it’s just crazy the level of depth and like nuance and forecasting that goes into it. We did spend hours and hours and nights and days and offsites working on and writing Europe 2031, but we’re not superforecasters and we’re not pretending to be. And so we said, let’s make it something that is very much grounded in research, but at the same time really leans into the storytelling element to just also cater towards a European audience.

In comparison to the audience of AI 2027, which draws, on average, on a more sophisticated pool of AI-pilled people because there just tends to be more of them in Silicon Valley than in Europe. We wanted to find something that allows people to slide into the narrative a little bit more easily, and you know, also cater to various degrees of AI-pilledness, or you know, kind of AI sophistication, if you want to call it that. Yeah, so that’s how this came about. And we had a lot of controversial discussions. We did max out a Google Doc. I didn’t even know that was possible, but it is possible to hit the upper bound of the character limit on Google Doc, and then we had to start copying things out of the doc and like into another doc because it was just like the amount of comments and like asynchronous.

Dan

Wow.

Judith

Back and forth, but then also, you know, kind of just sitting together around a big table for like several days and hashing out the story and, you know, going through every single point and asking, well, if we assume this thing to happen over here, we need to assume for this thing to happen over there. And I’m really, really proud of this. Obviously, you know, there’s a ton of things that we hope people criticize, and you know, criticism very much is the point of the piece that we put out. We actually were surprised by how little criticism we received. And we’re like, we would have been way more critical with what we have put out there.

But I think the thing that we really got right is despite having different opinions on certain aspects, especially when it touched more maybe the political dimension of what we were writing about, we always put the story first. We always said at the end of the day we want this to you know kind of significantly raise the attention level in Europe that people have for our AI future. And we want people to understand where broadly speaking things are headed and how underprepared we are relative to that future, such that we can create the urgency that is required to make a change. And whenever there were things where we disagreed, we always asked, Well, does it serve that point? Does it serve the overarching goal of why any of us are doing this in our free time and like you know, weekends and nights?

And if it didn’t, then we just let the argument go. We’re just like, you know, disagree and commit. It’s fine. And we did this several times over. And that was something that I thought worked so well. And again, how I wish a lot of our policymaking worked, where it’s not about, you know, finding some strange compromise that in the end is, you know, some kind of, you know, I don’t know, you know, grey-washed thing that doesn’t have any edge anymore. We either cut things or we took them to the extreme at the risk of upsetting some people who would have wanted more nuance here or there. Because we always asked, well, but does it serve that overarching point of why any of us are doing it? And so I’m really proud of us being able to put our egos aside when it came down to it.

Scenario or prediction?

Henry

So can I just press a little bit on this idea that it’s not a prediction, it’s a scenario, and I’d be curious to hear you say a little bit more about what kind of role you see scenarios as playing in the debate. Because you know, I can imagine someone coming along and saying, hang on, if it’s not a prediction, right, why should anyone update at all on the basis of this, right? I could write a scenario where Europe overinvests in compute and consequently gets sucked into some horrible economic bubble or contributes to ecological disaster, right? And you know, you can say anything in a scenario. So why should anyone update their priors based on something that is purely fictional in that sense?

Judith

Yeah. So, we didn’t write a Harry Potter story, right? So this is not just pure fiction in terms of, you know, whatever. I mean, very clearly we’ve got the compute forecast in there and again, it is a scenario that is grounded in well-researched facts, but at the same time there are narrative elements in the scenario and fictional elements in the scenario. And so what we basically want is to make sure that people know that we’re not taking our own work too seriously in the sense that, you know, we’re not claiming that this is 100 per cent falsifiable or that you can trace it step by step.

Like we know that the world has a lot of complexity, a lot of gray space, a lot of nuance, and we just want to have respect for that and you know, not make it seem like, you know, we’re kind of some type of predictive fortune-teller who can see into the future. I actually think, Henry, you know, there’s a lot of critical voices in the German government and in every government saying that AI is a bubble and you know, all this compute is gonna be a waste and we’re all gonna be sitting in like dark data centers because no one needs any of the GPUs that we’re putting in there.

I would love for people to actually write a narrative scenario about why that is, because it forces you to sit down and follow the logic of your own arguments, not just at a shallow, this is my hot take, you know, I’ve had a beer and we’re kind of chatting, or like, you know, a journalist asks me a question and I’m just able to give you one sentence without any substance. Having to write, you know, 50 pages of how that scenario actually plays out and the types of assumptions that allow you to then uphold it in the future. I would actually love to read that work because I think if people were to engage in that work, they would understand how flawed much of the very shallow level thinking that too many of us are still engaging in is.

And so, you know, I think the narrative scenario allowed us to have these super long conversations, to be in this ultra-long Google Doc and to just try and figure out one version of what the future could look like, while knowing that there’s many versions in terms of how the future could play out, but it forced us into very, very concrete and somewhat logically consistent thinking. And to all the critics out there, you know, that think AI is overhyped and it’s all overblown, please, I would love to read your scenarios and your logically consistent thinking about why that is, because it will allow me to write a really substantive rebuttal about, you know, kind of giving you the counter-arguments. But yeah, open invitation to a lot more scenario work to anyone who feels, you know, kind of energized to do that.

How fast will AI progress?

Dan

So in this scenario, Europe, European elites, policymakers, civil servants and so on, they mess up by underestimating the speed of AI progress and the magnitude of the changes brought about by this progress. And those are two of the kind of core assumptions of the document that there has been rapid progress, that progress is going to continue. These technologies are going to diffuse very quickly throughout the economy and society and so on. What do you say to people who are skeptical of those two assumptions, both about the rate of progress going forward, but also about the magnitude of the changes that are coming?

Judith

Yeah, so I will say two things, or three things. I think there’s the question of how quickly the exponential is going to progress, and then how quickly it is going to diffuse into society. We are absolutely well aware that those two things are not the same thing. AI is developing exponentially and people change maybe linearly if at all. You know, most people don’t like change whatsoever, and we’ve just built quite complex systems in terms of organizations, but also the way that society works, where change cannot just magically happen overnight. We are absolutely very well aware of that. In fact, this was one of the points where I would say in the author group, we had the most discussion, especially around diffusion. What does it mean for like economic growth assumptions?

What does it mean for the assumptions around, well, you know, even if we don’t have the most advanced models, maybe, you know, if the models are significantly better than the models that we have today, does that still mean that relatively speaking we’re better off economically than we are today? Like there’s a ton of nuance and there’s so many amazing economists. I’m trying to keep up with the amount of great work, even though I still think we don’t have enough great economists working on this, you know, given how important of a subject that is, but like we’re aware of like how complex modelling that, you know, kind of really is.

And again, this is just like one assumption that we took in order to like break down complexity because we didn’t want to put a bunch of econometric data and forecasting into the scenario; that would have you know kind of defeated the point of making it something that is like light and like easy to digest for as many people as possible. Lots and lots of discussion within the author group around like the exact assumptions that we take here. Now, at the same time, having said this, two things. So, before we published, we had a bunch of experts and people that we respect really deeply from AI research and AI policy research, just read this and like pick us apart and give us like their harshest criticism.

The most consistent piece of criticism was that we’re actually underestimating how quickly AI is progressing and like especially a lot of AI researchers basically said that a lot of the stuff that we’re assuming in 2027 is very much already happening in labs in 2026. So a lot of the stuff around recursive self-improvement and so forth. So, by and large, I would say expert feedback was your timelines are like too long, you should pull them back by a year. And we were like, we could do that, but A, it would involve a lot of rewriting, and B, you know, kind of in comparison to maybe also AI 2027, some of the more authoritative safety work. This is not about like the exact, you know, kind of being right on timelines.

This is more the overarching kind of policy story and you know, kind of industrial story for Europe that we wanted to tell. And we felt we could still tell that, even if our timelines ended up being a little bit longer than the ones that experts had. First feedback. Second feedback, diffusion into society. Who knows, right? I mean, this is one of the points I would love for more discussion on this. We see it as part of like our own, you know, kind of organizations in terms of like AI adoption. I see it as part of my job as, you know, the co-CEO of Langdock, where we now have more than 10,000 customers and we’re very much kind of the portal through which they adopt AI. Like we see all variations of the rainbow, right?

We see, on the one hand, you know, companies that are very advanced when it comes to coding agentic use cases. And we see other companies—I was just speaking to a massive company in Germany yesterday who are now talking about a first timeline to introduce AI chat to the employees, right? And it’s 2026, and like some people are already, you know, some engineers are already like running their lives on autopilot with a lot of agents, and like here, you know, is another company that is just talking about like how do we even introduce this like AI chat thing to our employees. And both things are true at the same time, which I think is what makes this so complex. I do think increasingly we’ll live in kind of different times at the same time.

Like there’s gonna be some companies that are gonna be ultra-advanced, and some teams and some individuals, and then there are going to be some—and probably not just some, but lots and lots of companies that are, you know, not that advanced and that are very much still on their AI adoption journey. I heard one of the CIOs of one of the biggest companies in the world in the FMCG space saying that in 2030, you know, many companies will still be beginning to go on their agentic transformation journey. And I think he’s right. It’s just the reality of how slowly people in the end adapt. But what that then concretely means for this diffusion piece, and what exactly the form factor is. Today we’re obviously working on the back of the form factors that we know, which are mainly chats and agents.

Maybe there’s gonna be additional form factors that are gonna make diffusion a lot easier. Maybe it’s gonna have to do with hardware and so on and so forth. We already see kind of the timelines of diffusion of technology, you know, rapidly accelerating. If you look at, you know, any past kind of application of technology, you know, that time to reach a hundred million users or you know, then a billion users, you know, like AI is already, you know, kind of smashing through all of those records.

So I don’t think it’s inconceivable that, you know, with one more form factor and continued acceleration of AI progress, the systems are gonna get so good that you know suddenly they’re just gonna be everywhere and we’re gonna be, you know, interacting with them in a very natural way that’s very different to still a lot of the like harnesses and a lot of the plumbing that is required to make AI truly useful in organizations. And so not being able to completely account for that diffusion piece of AI, I think that’s why we chose the type of diffusion timeline that we used, where we basically said, you know, it’s meaningfully gonna affect competitiveness. Not having access to the same type of powerful AI systems is gonna make you a meaningfully less competitive company than if you do.

But we didn’t assume, you know, kind of an economy completely run by robots and you know, one where human labour no longer has any worth and it’s like all dominated by AI either. And so I think that was kind of us trying to strike some sensible middle ground, very much accounting for you know, kind of an underlying distribution in terms of like how exactly each of us think about this diffusion part playing out in real time.

Is AI a normal technology?

Henry

I have to say I really did enjoy having a scenario that steered this middle line between taking AI quite seriously, but also wasn’t just well, basically we hit some kind of transformative point in the next three or four years and you know, who knows what follows after that. Dyson spheres around the sun, etc. So I also think it’s helpful. Out of interest, Judith, where do you fall on the kind of AI as normal technology framing of what we’re going through?. Is this a framing that you would endorse? Because I mean, one thing that we’ve discussed in the show in the past is that even on the AI as normal technology framing, this might still be the most consequential technology of certainly our lifetimes, maybe, you know, the last hundred or hundred plus years of human history.

Judith

Absolutely. You know, the very honest answer is I don’t know. If I told you I know, I would be bullshitting. I don’t know. What are the things that I do know? The co-founder of my company, Lenny, has this beautiful saying where he tries to not think about the things that will change, but he tries to think about the things that won’t change. So, what are the things that I know won’t change? People will still be people.

We’ll likely still want to engage in some meaningful way of you know coming together and engaging in something that gives us a feeling of contributing something valuable for the people around us that may be called work, that may be called something else 20 years from now, I don’t really know, but I think many of the structures that we formed societally and how we engage with one another and how we kind of come together to work on something that is bigger than ourselves, that will remain. Having said that, you know, and working in Germany with you know many companies that have been around for hundreds of years and will likely still be around for hundreds of years, you know, I just don’t think that many things on the margin, you know, kind of change that rapidly.

I was just saying earlier that I was on the phone with a big company in Germany, two and a half thousand employees yesterday, who are now thinking about sketching out the journey of how they can start introducing AI chat to their employees, right? So it’s 2026 and some people are running their lives on agents, and here’s this company that is like starting to use AI chat. And there are many, many more such companies out there. So I think overall humans will not change as fast as we think they may. But at the same time, let me tell you a really interesting story. What got me thinking was meeting one of our customers, and he is working at a big chemical company and actually the third generation of his family to work there. So his grandfather already worked there, his father worked there, now he works there.

His grandfather worked there as a locksmith, his father worked there as a chemist, and he’s now working there leading AI transformation. He has an 11-year-old daughter, and both of us wondered out loud what his daughter may be doing working there in the fourth generation. We both were pretty aligned on it not being AI transformation, but like some completely new job scope that you know probably doesn’t make any sense to us just in the same way that whatever it is I’m doing doesn’t make sense to my parents and certainly not my grandparents. And so we’ll continue seeing kind of change over the course of, I would say, generations, and that change is pretty strong, but I think over the course of just a couple of years, I think there’s many, many things that are actually gonna stay the same.

Now, having said this, which is more kind of specific to jobs and how much change are we gonna see and how much change like organizational and societal systems can allow and so on and so forth, I think there’s just a second piece. And It’s so difficult for me to size the shape of this piece correctly, because I feel like I’m underestimating it and overestimating it at the same time, which is: what even is the value of intelligence? I think on the one hand, we still vastly underestimate the power of intelligence because we only know human intelligence, right? Like I know I’m kind of somewhat smart, and Henry and Dan are way smarter than I am.

Like I’m able to judge that differential, but like overall, you know, none of us are like ultra, ultra, ultra, ultra smart in the way that, you know, kind of a supercomputer can do certain computations. And so just being able to correctly understand and value what intelligence beyond human levels would mean at scale—I find that very hard. Like what’s the market size for that? I have no idea. You know, what’s the market size in terms of drug discovery, security, creativity, anything that’s open-ended? I don’t know. I think it’s gonna be damn valuable, and I think we’re just still vastly underestimating just how valuable smashing through human-level intelligence will be. One perspective. The other perspective is, you know, there are many more things that make up our day-to-day interactions and our lives than intelligence. Like intelligence is but one input factor to the human experience.

There’s many other input factors that also hugely matter. And of course, there is this perspective of like if intelligence is abundant and it’s available on tap, it’s just gonna move into the background and whatever becomes scarce. I loved Alex Imas’s piece on like you know, kind of what is scarce in the future, and this whole piece around the relational sector, and you know, kind of the relationships that we engage in as humans, that’s the thing that’s going to be scarce and that’s going to be much more valuable.

So I think things like the in-person, real-life experience, the charisma, the physical embodied nature of our life as humans, the fact that I was never made to be sitting and staring at a desk all day, but I was made to probably like dance and engage in some type of, you know, kind of physical interaction because that is literally why I have this body, not to sit on my bum all day. Like I think there’s gonna be all sorts of like rediscovering the value of human life beyond intelligence. And we’re also underestimating that piece. And so sizing those two things together is like uniquely hard. Yeah, so that’s probably why I don’t really know.

I’m just curious and just updating my system with like every new model and kind of just trying to engage in observational analysis about the world around me, and trying to understand what exactly that means for the type of future that we’re headed in.

Institutions, incentives and adaptation

Dan

Yeah, I think that question of what’s the value of intelligence is so important. But I think not only is it true that there are other things that are important, sort of independent of intelligence, but it also seems like the extent to which intelligence will be important depends on the structure of our institutions and our organisations, and that there are circumstances which I think sometimes get kind of underrated in importance where unleashing more intelligence of a distinctive kind can actually make things more inefficient and more dysfunctional. So academic research, which is the domain that I know kind of most about, you would think intuitively that giving everyone access to this extraordinary AI-based intelligence would just unproblematically improve the quality of research, improve the ability of people to make scientific advances and theoretical advances and so on.

But there’s another scenario here where unleashing this intelligence actually just ends up clogging our journal system with lots of crappy science. It means that academics offload too much to these AI systems. So they actually kind of disable some of their critical faculties and so on and so forth. And that’s a scenario in which you do have kind of rapid AI progress and the capabilities are really, really important, really powerful. But because our institutions haven’t adapted very well, it doesn’t necessarily translate into kind of tangible progress in the real world. And I take it to kind of steelman the AI as normal technology perspective. The argument there is you really have to focus on those organisational, institutional incentive structures and norms and procedures and so on to really understand how it is that AI is going to have impacts on the real world.

You can’t just focus on the capabilities of the technology itself. So I guess one question would be if that kind of perspective, the AI as a normal technology perspective, is correct in saying look these issues of diffusion and institutional adaptation, they’re very complex, there could be real kind of slowdowns here relative to expectation, if that perspective is correct, would that materially change how you’re viewing this whole area and would it materially change the recommendations that you’re making when it comes to European policymakers and civil servants and elites more broadly.

Judith

Yeah. So I think what I would say is that I think you’re right in the analysis of, you know, kind of the academic system and us relying on AI too much such that it actually leads to negative outcomes and not the positive outcomes that we were hoping for. I think there’s plenty of examples from the past where this has happened. I mean, you know, internet technology, social media, arguably the fact that, you know, all of us are, you know, all of our backs are like absolutely fucked because we sit again at a desk all day, and you know, this whole sedentary lifestyle is not actually all that good for us, even though it brought us all this progress and flourishing. And like white-collar jobs, and so on and so forth. So I think there’s like plenty of examples here.

So on the one hand, I do think we have a duty to drive change with confidence, but at the same time with the type of foresight and responsibility that allows us to land, you know, kind of in the best possible trajectory for society. So just this like let technology rip, Silicon Valley, you know, like we’re gonna clean up the mess after, but like the mess is gonna happen anyway, so let’s just let it rip and you know, kind of I just don’t believe that. I think it’s more convenient, it’s easier, but I do just think, you know, humans have evolved. We are able to assess the world. We kind of have a broad understanding of what makes a good life, what makes a not so good life. We have certain values that underpin our society.

We should be able to engage in the kind of discourse and the kinds of guardrails that allow us to introduce technological change alongside the types of systems and the types of education that enable us to land in the best possible outcome for humanity, such that the cost of an innovation, because there’s always a cost and there’s always risk that comes with innovation, is minimized relative to the upside and the benefit that society can reap from it. Now, what I think is really important though is that there won’t be the possibility to drive change without at the same time engaging in some type of risk or some type of discomfort that just comes with, you know, kind of any unit of change just being a unit of change.

It’s not the world staying the same, but by and large, people end up being better off than they were in the past when they, you know, kind of push through the friction of that change. But I think both thoughts need to be held together. And I think there’s a somewhat unhealthy tendency to want the upside of progress and the upside of change without wanting to engage with any of the disruption and any of the change. I think very much that is the conundrum that Europe is in right now. I always call this the two Ps. Europe is a system of principles that I actually think are the right principles: self-determination, human dignity, freedom, peace. You pick your choice of like your European value. And then we formed procedures to protect those principles. The procedures, however, were always meant to serve the principles, not the principles serving the procedures.

We are now in a world where all principles are simply serving the procedures of a world and of a system that is maximally optimised for downside protection without actually being able to reap the benefits that, you know, engaging in change, engaging in progress actually enables. And so I think we need to in many ways do away with the procedures that we formed that serve a world that we no longer live in such that we engage in more friction—what we call hard trade-offs in Europe 2031. There is this idea that we get to reap the benefits of all this.

You know, this whole idea of like let the US and China invest all this money into AI and we just get to reap the benefits as consumers, but we don’t actually need to invest in any of the capital-expenditure build-out or like let them build all the data centers and we just get to use like that is just naive. It’s wanting all the good stuff without engaging in the hard stuff. I just don’t think that’s the world that we live in. And so what are the trade-offs that we’re willing to lean into that will hurt, that will mean that there are things that we’re giving up, there’s you know, kind of things that we fought for that we may have to walk back such that we can overall land in a better place for everyone. I think that’s how I think about it.

And then it’s basically engaging in the kind of discourse and sense-making to determine exactly what that means in every single, you know, kind of case. And we can talk about the policy recommendations of the piece because we have thought about this long and hard and that’s kind of where we landed and much of it rests on the back of like hard trade-offs, in order to get to like a much better place for everyone. But that’s probably how I think about it.

Creative destruction and the Fable episode

Henry

I actually really like that framing, Judith. And it reminds me, maybe I’m free associating a bit here, of Daron Acemoglu’s sort of notion of creative destruction as being key to growth and progress within pluralistic countries or pluralistic societies, where you know you can recognize that there are enough enfranchised people able to reap the benefits of new technologies, that there is that kind of broad enough coalition to allow for planned but ultimately disruptive change in a way that allows for compensating losers and so and so forth. But yeah, Dan, for what it’s worth, I also strongly agree with what you were saying about the importance of incentives and institutions.

And I was just reminded of something that I’m seeing more and more of, both from hearing from friends and seeing what’s discussed on Twitter, is that just something as simple as our recruitment procedures are increasingly just collapsing because like the costs of sending out 10,000 CVs, right, has plummeted to zero. And like, for example, one obvious, I realise controversial, solution to this is to allow companies to charge even like a totally nominal dollar fee for job applications. But in many jurisdictions, like I believe California, that’s currently illegal. So like trying to figure out however falteringly and fallibly, the right kind of changes to our laws and institutions to deal with a very different world, I think, yeah, it’s one of the absolute challenges ahead.

So Judith, I do want to come on in a second to asking about the actual prescriptions of the report, namely these ideas of special compute zones and building out European data infrastructure. But I did want to flag, I did want to ask you about one quite interesting and fun maybe coincidence that happened, like basically in the days immediately after releasing the report, which was that we started to see transatlantic crackdowns on access to models, namely in the whole Fable saga. So very, very roughly, for anyone who’s not aware, shortly after Anthropic released Fable, the Trump administration asked them to limit it to US citizens. Anthropic basically said, we’ve got no way of doing that. So they rescinded Fable access to everyone for a couple of weeks. And on the one hand, I think I tweeted at the time, it’s like, this wasn’t supposed to happen until 2029 or 2028.

So it looked on the one hand like a really far-sighted prediction of the report that happened to coincide with the week of its release. But on the other hand, after I thought about it a little bit more, maybe there were some nuances of difference there. So, for example, the fact that Anthropic really couldn’t figure out a way to limit it just to US businesses or US individuals. So yeah, I’m curious if you have any reflections on that whole little saga.

Judith

Yeah, no, absolutely. So I think it’s so interesting because it isn’t what we were describing, you know, kind of word by word in the scenario, yet people kind of interpret it as that, you know, suddenly we’re fortune-tellers, because like that’s exactly what Europe 2031 was warning about. But I think this shows how much complexity gets lost. You know, I mean you read a story and it’s a long story, and in the end you take like high-level messages away, and then the thing happens and you’re like, isn’t that exactly what I was just reading about? Yeah, not quite. It was quite a different context and quite some nuance.

But I think by and large, and this is the much more important point, you know, I think the big piece that Europe awoke to was there is a technology that is very powerful and the US may ban access to this technology on the basis of whether you’re a US citizen or not. I think that is the high-level point. Many people called it a warning shot. I do believe it was a warning shot and it is a warning shot that in a different form, I mean in our scenario it focused on like compute restrictions, right? Like the US, I mean we also spoke about the White House executive order and kind of you know kind of screening of AI models and that continuing.

But then there is the big compute crunch that we outline around 2029, where basically there are these three zones: tier-one, tier-two and tier-three zones, where basically, you know, there’s such powerful models. The US, you know, is now screening access to those models, is withholding access to the best models, to kind of the domestic market until the wider market gets access to it.

And then on top of that, because it’s so expensive to serve these models and because there’s so much demand for these models, and we don’t have enough compute, and most of the compute is on US soil, the US is therefore also restricting or kind of rationing compute so it gets more expensive, and the compute quota that European countries get is no longer one that actually fulfills the real demand that they have such that everyone basically doesn’t have enough compute, which in the end just means the ability to run inference on these models, so to use these models, which puts us in a materially worse situation than the US. That is the scenario as we kind of, you know, wrote about it in Europe 2031.

And I think you can interpret the Fable incident as some variation and some nuance of that overall concern, even though as you said, you know, it was kind of on the back of a lot more nuance. And I will also say, and we said this on the back of this happening, you know, there will always be security and safety concerns that we all need to take very seriously, European countries and the US alike. And so, you know, kind of the 19-day ban and then the jailbreak fix that Anthropic put out that allowed them to restore access to the model for everyone. We will, you know, kind of always champion a world in which the models that we put out into the world are, you know, kind of in the safest form possible, and that may mean that you know certain types of screening are required.

What we worry about is a world in which the US gets to use that as a means of exerting pressure on other countries, or just a world in which the access to these models for other countries is so delayed or rationed in some way, shape, or form that actually, economically speaking, as well as security speaking, you know, just puts us into a worse position relative to other countries.

Why Europe needs compute

Dan

So as you just alluded to, Judith, the main policy recommendation of the report is basically to enable massive investments in compute and building data centers on European soil. So first question, why is that the recommendation? Could you unpack that in a little bit more depth? And second question, why is it superior to another recommendation, which would be, can’t Europe and other middle powers band together and just build our own frontier AI?

Judith

Yeah, so compute. Today, Europe hosts about 5% of the world’s compute. The US hosts about 75 to 80%, which means that we already are very dependent on the US for kind of European compute needs. We want a world in which Europe has a real seat at the table and has the ability to take care of its own AI transformation needs to the greatest extent. That for us does not mean a world of you know kind of sovereignty as independence, but sovereignty as interdependence, which we do believe you know needs to some extent rely on transatlantic collaboration with the US, but also rely on collaboration with other middle powers. I’m gonna try to unpack the compute piece separately from the model piece. For the compute piece, as we tried to describe in the scenario, we all do worry about a future in which the world will be very compute constrained.

I think if you just look at the world today, and again, you know, as I’m speaking to lots and lots of companies who are still in the early, early, early innings and early journey of truly making use of AI. We just don’t live in a world where all of our lives and all of our organizations are already run by agents. And so you might think we couldn’t possibly need more compute than we already have. I think we’re still vastly underestimating that if this technology continues to progress and if the kind of you know transformation and economic potential that we gain from it is one that we think is going to be powerful, there’s gonna be vastly more compute that we need, and we can’t even you know begin to picture what that will mean in terms of having to put real metal on the ground.

Compute is also such that you can’t just, you know, it’s not like a Big Mac. You can’t just, you know, like, okay, you know, I’d just like to have two more. Yeah. There are real lead times when it comes to lithography machines and chips, and then to actually building the data centers, the permitting, the grid connection, the powering and so on and so forth. And so we think it is really important to build this very, very quickly now, with foresight into how bottlenecked we will be once the true power of agentic AI, or powerful AI, is unleashed. I don’t think we’re yet in the midst of that. You know, we haven’t even begun to understand what that will look like. We think this is really important. Why is it important to have compute on European soil?

Because if the compute is on European soil and primarily serves the European market, we think it’s gonna be significantly harder to shut us off: we’ll be a very, very significant part of the market. You know, the compute and the data centres are actually going to be on European soil. So even if the US were to turn that off, that’s a massive investment that you know is kind of gonna be some cost and that is gonna hurt. At the end of the day, you know, we’re all also economic actors, we have economic interests, and so you know, of course, there could be a world in which the US is just like, no matter what, we will still cut access to everything for everyone. We just don’t think that’s how, you know, at least somewhat rational economic actors act.

You know, they will act, you know, based on the total picture of their economic interests. If a large part of the compute and of the market is being served by this compute in Europe, it’s gonna hurt meaningfully more to cut access to this compute and the models that are being served here. Now, having said this, there’s a really big pushback that we’ve received around this whole notion of like why should hyperscalers be the ones building this compute? And I want to say one thing. We want compute on European soil. We’d love for sovereign European players to be building much of this compute. We’re all for it.

We’re just also realists when it comes to, again, the hard trade-offs that we need to face when it comes to not just the theoretical window-dressing of, yeah, we’re engaging in some type of compute strategy for Europe, but the actual question of whether we are significantly scaling the compute supply to what we think we need: around 15% of the global compute in 2031, which will be around 50 gigawatts, which is something like an order of magnitude bigger than you know even the most ambitious kind of European build-out plans. So that’s where we think we need to land. And if we want to engage seriously—the German Chancellor was on stage a while back and he said we want to triple compute in Germany over the course of the next five years. Well, but tripling is not nearly going to be enough. Tripling we will likely be able to do with sovereign players.

But you know, tripling and then ten-xing the tripling, it’s gonna get very, very hard. Why is it gonna get very hard? Because financing compute is supremely expensive and supremely complex. What we need to not lose sight of is that the fact that the US is able to build compute at the scale at which they’ve built it is on the back of a very small number of companies with the types of balance sheet and capital-expenditure capabilities that we just lack in Europe. There just is no structural, you know, kind of counterpart to the Magnificent Seven or, you know, kind of the types of efforts of XAI, Meta, Google, Amazon, and so on and so forth.

And so, in the absence of those types of players who can engage in that funding and can basically, you know, kind of even buy up future supply chains, buy up future, you know, kind of compute capability that can be built, we need to work with the players who are already dominating the space and who have the financing capabilities, first argument. There’s a second argument around well, if we have sovereign compute players in Europe, and again, we’re all for it, we want to build, you know, as much sovereign compute as possible. We’re just realistic and facing the trade-off of that not being enough to kind of serve the European compute and inference needs.

There’s a second argument around the sophistication that the sovereign players need to show in terms of their technological capabilities that would enable, you know, the likes of Anthropic and OpenAI to actually share their weights with those players. Today, you know, the only way to consume the weights of these providers is through the hyperscalers, through, for example, Azure and Frankfurt. It’s not through, you know, kind of the sovereign, more maybe nascent or in some ways, oftentimes unfortunately, technologically less sophisticated players that we have in Europe. And so also accounting for this gap and for the fact that as these models get more powerful, the types of security and overarching technological hosting capabilities will only grow. And so the players who already have proven that they can do that are the hyperscalers.

So again, it’s just the pragmatic perspective of if we want to get to 15% of global compute, which is still smaller than Europe’s total economic footprint in the world but is, we think, roughly the right ballpark, we need to go to where the centre of gravity around compute build-out is, which means that this is not going to be net-new capacity that we’re just going to add to the compute market. The compute market is already maxing out; it’s operating at its maximum capacity. So it’s going to be about diverting capacity and making the economic case for the hyperscalers: Europe is a great market that you want to be serving and already are serving when it comes to enterprise demand. It can be a great market in which to continue building data centres and diversifying overall compute capacity across more regions.

It’s gonna make for a great business case for those hyperscalers, and at the same time for any sovereign player who also wants to engage in compute build-out. We’re all for that. We’re just facing the hard realities of what is required to get us to 15% of global compute.

Henry

So one piece of pushback I’ve heard on compute goals—well, actually, I’ll give two. So the first piece of pushback I heard on this suggestion is just that Europe is not the right place to try and compete on compute. We have some of the most expensive energy in the world. We have very expensive land, labor is quite expensive. There are things we are really good at, right? Like high-end science and engineering, but like compute is probably not the right battlefield for us to be competing on, is one argument. Another argument I’ve heard is, and you know, apologies here for spoilers for a dramatic event late in the story. But obviously ASML plays a really critical role towards the end of the story, where Europe is basically unable to leverage ASML’s critical role in the AI supply chain.

And so the argument here is: look, Europe already has, like, you know, if you’ll forgive the vivid image, its boot on the neck of the AI supply chain. Actually, in more than one place, not just ASML, also Zeiss, right? So if Europe is unable to leverage Zeiss in this world and it’s unable to leverage ASML, why think a whole bunch of basically commodity-level data centers are somehow gonna tip the balance in our favour?

Judith

Okay, and I also want to come back to the model question of why shouldn’t we just instead engage in like building frontier model efforts in Europe as well. So three points to come back to. So maybe the first one with Zeiss and ASML. We do also speak about this idea of middle powers where, you know, Germany has Zeiss, ASML is in the Netherlands, and Japan has some really important inputs into the semiconductor supply chain. South Korea virtually produces all of the world’s memory—or around 80% of it. So there are a lot of middle powers who are structurally maybe in a very similar situation to the one we are in, where we hold or control important choke points.

And we do absolutely believe that it is sensible for those middle powers to come together and to engage in you know, alliances that allow all of us to strengthen our joint seat at the table and kind of, you know, stand up for the frontier-model access or types of sovereign demands that allow us to strengthen the position of, you know, kind of the citizens that we represent. We’re all for it. At the same time, in a world in which the world is compute constrained, we see it right now with Kimi K3, which just came out a few days ago and already Moonshot, the company behind Kimi, is constrained in terms of being able to serve the model just because there’s so much demand they don’t have the compute. To me, it is like a completely nonsensical argument unless all economic considerations go out the window.

In a world in which compute needs are gonna be real and there’s gonna be massive compute on European soil, it would hurt massively to just like let the compute sit idle and be like, we don’t care about like, you know, the massive amounts of racks that could be serving this like economically really valuable thing. That just makes no damn sense. I’m sorry, I just don’t believe that’s the world, you know, that we live in today. And that’s certainly not the world in which the thing that the compute is serving is only gonna grow more valuable. And therefore the demand for it is gonna explode and we’re just gonna say, like, yeah, let’s just ignore all this compute over there. I mean, it is almost as nonsensical as Germany getting rid of its nuclear power plants in the face of really, really high energy prices.

I’m not saying that there aren’t situations in which countries engage in like massively stupid overall decisions. Like it does unfortunately happen even to the best of us, but by and large, the argument just makes no sense to me. Compute is going to be really valuable. It is not just, I’m sorry, it’s not just a commodity, you know, metal thing that you’re putting in the backyard somewhere. Building data centers, being able to engage in the kind of parallel computation required, and the computation required both to do training and to do inference is supremely complex. It’s not just something that like everyone and their mother can be doing tomorrow. In fact, as I just said, we have some neoclouds that are doing a great job in Europe that are very much trying to build, you know, kind of sophisticated data centers and sophisticated compute offerings. But this is hard.

This is not easy, not just on the build-out side, but also on the operating side. You know, this has a lot to do with like downtime and you know, kind of making sure that, you know, if one node is down, you can kind of replace it. I mean, I’m not a compute expert, but by and large, this is a very, very complex field. And so I just reject the notion that the thing that is gonna be extremely valuable, we’re just gonna let it sit idle and we’re just not gonna care about it. I think it’s the same as if you thought about electricity and you said, ‘Let’s just shut down the grid. Let’s just make everyone go dark.’ Who needs electricity Anyway? I think if you switched off electricity for people and they could no longer use their fridge or their hairdryer, they’d be revolting like on the street tomorrow.

I think that’s a little bit more similar to the type of world that we’re gonna be in. It’s just that because AI is for many people not essential to their lives yet, because we’re still in the early phase of the exponential and the transformation as it’s gonna unfold, it’s hard for everyone who isn’t already relying on Fable to picture this. But for the people who relied on Fable, for all the developers, I mean, I used Fable for a couple of days before it went down. I loved the model. I thought it was like an absolutely beautiful model. It was the first model that I thought I had some type of like wisdom experience with.

In a way that I didn’t with previous model releases, I really felt the step change in capability, even though it’s the lobotomized version of like the actually capable model that like people at Anthropic get to use internally. So I just you know, the people who were very much using it felt the model being withdrawn. And that is 1% of the type of world that we’re gonna see as more and more of our lives and the things that we take for granted rely on this. So I think we need to do a better job in storytelling because I think this point has just not come across in the way that it should have come across. Back to energy prices. Why should we build compute in Europe?

Our energy prices are a mess in Germany and in many other European countries, but we have France, we have the Nordics that also already host, you know, the vast majority of European compute. We have the Iberian region where we actually do have, you know, more abundant and cheaper, more competitively priced energy. We also in Europe are, generally speaking, you know, kind of world leaders when it comes to many of the kind of renewable technologies, you know, that we’re that we’re putting out there. So like I think in theory, this could be a massive opportunity for Europe to really lean into, you know, one of the things that we want to be doing Anyway, which is accelerating green energy build-out. There are certain areas in Europe where hosting compute will be significantly more competitive.

But even in Germany and even with kind of, you know, less competitive energy prices, the overall productivity that we’re able to gain from compute and also the overall margins that you know the very profitable data centers are charging are still such that they allow us to actually, you know, run these data centers profitably, even in regions that have structurally less competitive energy prices. I’m not advocating for that being a good thing. I think it’s a bad thing. I think we should be doing everything we can to bring energy prices down.

And having said that, we should also bear in mind, and this is me speaking as a German, that, you know, kind of the wind energy build-out that we’ve been engaging in over the past couple of years was faced with a lot of backlash from society because I think people were not taken along this journey in the right way, and they couldn’t share in the gains of wind energy in the right way. I do think, you know, building data centers should be done with maximum consideration for the communities around these data centers in mind, you know, allowing them to share in the economic gain and to, you know, kind of really do this kind of hand in hand with society. A lot of this is down to storytelling, where again, I think the story is very different.

It’s just a bunch of racks, you know, in a data center with a bunch of cables that we connect. This is not a story that’s gonna excite anyone. But the story isn’t exciting if I talk about the energy grid. Who gets excited about the energy grid? I certainly don’t, but what do I get excited about? All the things I get to do today because of the energy that comes out of the grid. Are you kidding? That’s insane. It’s absolutely insane. If I compare that to life in Germany or in Europe or anywhere in the world 50 years ago or 100 years ago without energy, like it’s massive the type of transformation, the type of progress that we’ve benefited from because energy grids exist. Energy grids themselves aren’t sexy. The things they enable, very, very sexy.

So what we need to do a better job of is telling people exactly how their individual lives, as fathers, as mothers, as families, as teachers, whatever it is, are gonna be positively impacted by the fact that we build these things that don’t look very sexy, that are power-hungry, but that actually provide massive opportunity for a very, very broad base of people.

Should Europe build a frontier AI lab?

Dan

I think your answer to this is clear from what you said already, but just to be really explicit, why should we not just try to build a frontier AI company that can compete with OpenAI and Anthropic? Okay.

Judith

Yes. Okay, sorry, let’s get into this. So we didn’t include it. What we did include also in the Economist piece where we wrote about kind of Europe 2031 is we do think that any effort to—sorry, okay, let me get this right because it does require some nuance. So there’s a reason for why we didn’t lead the report with Europe should just build a frontier lab. Full stop. That’s the recommendation. Why didn’t we say this? Well, we didn’t say it because we do believe we’re not early in the race anymore. We don’t therefore think it’s too late, but we just want to be mindful of the fact that there’s certain things that have a less binary success-or-failure outcome than building a frontier lab where like either you succeed at building something that competes with OpenAI or Anthropic or you don’t.

And unfortunately, the European structure and the kinds of struggles that, you know, kind of labs like Mistral have seen, but also just the broader regulatory environment around even being able to train foundation models in Europe, because even if we had the compute, all the compute for training, unfortunately it’s still regulatorily speaking, much easier from a fair use perspective to train foundation models in the US than it is in Europe. It’s just we don’t have the environment that has lent itself well to building these like massively capital-hungry, you know, ultra-ambitious efforts at a point where we’re already—I never want to say too late, because I don’t think it’s too late, but like where we’re no longer early in the race. We still think it is sensible for Europe to likely try and engage in this.

If we try and engage in this, we need to do away with this whole idea of building something that can compete, you know, kind of with the Chinese model. So, like frontier minus six months, frontier minus twelve months is the wrong ambition. If we do engage in this effort, it needs to be like a Manhattan-style effort, it needs to be ultra-ambitious, and it can’t be, let’s fund it with two billion, it needs to be let’s fund it with like 50 billion. Like it needs to be a proper pan-European, ultra-aggressive effort. We don’t right now see this effort, or the seriousness, coming together. Otherwise, I would definitely have recommended that as one of the kind of policy recommendations towards the top.

Now, at the same time, and I’m gonna go into it in a second, people are getting very confused, or I think carried away with like frontier versus non-frontier, and either, you know, kind of either everything, all the value accrues to the frontier or all the value accrues to like frontier minus N. I think both are wrong. I think value is gonna accrue to both sides of the spectrum. Like, yes, it is true that a lot of economically valuable tasks do not require Fable-level capabilities. Kimi K3 launched a few days ago. It’s an absolutely astonishing model. Even Nemotron, the open-source model from NVIDIA, is a really capable model that was trained entirely on open data. Like there’s a bunch of models, many of them open source, that are, you know, more efficient than the frontier models. Maybe not as capable on all fronts, but still very, very good.

And I think that’s great news for any business owner, that’s great news for any consumer, because it means you’re gonna get, you know, good intelligence for like a much cheaper price. At the same time, I don’t think that negates the need for Europe to negotiate access to whatever frontier exists. And I’m just mindful that any frontier effort, no matter how ambitious it is, And unfortunately I don’t think we’re in the right mindset and regulatory environment to make this an ultra-ambitious effort, which it would need to be to stand a chance. But even in a world where we don’t succeed, we need to have enough weight at the table because we’re hosting the compute, and because we’ve engaged in middle power alliances to negotiate access to the frontier that already exists.

Because a world in which there is a frontier that Europe doesn’t have access to puts us into an adverse situation with regard to our security, with regard to you know even things like drug discovery, I mean all sorts of things that we would want to benefit from on behalf of our population that we won’t have access to, that you know, other nations are gonna have access to, that I just quite frankly don’t feel comfortable with. And so we formulated the policy recommendations in Europe 2031 from the perspective of even if we don’t manage to build a frontier model effort, how can we still be in a situation where we have enough weight at the table to negotiate access to the frontier that already exists? That doesn’t mean that it’s not important to have a frontier if we could build it.

And I hope that people take this effort very seriously, but we think we need to prepare for both worlds, one in which we do have an Anthropic in Europe and one in which we don’t.

Henry

So that’s super helpful in giving me a sense of the overall arguments. And actually one thing you said in your answer to the previous question clicked for me. Just confirm for me if this is right. So one problem with sort of using ASML or Zeiss is that it’s kind of a suicide pact, right? They are as much dependent on NVIDIA as NVIDIA are dependent on them, ultimately in terms of supply chains. Whereas compute is a more flexible resource. It is valuable and can be redeployed for different purposes. So by building this kind of large compute reserve, it gives Europe potentially leverage that doesn’t come down to sort of brinkmanship. Is that part of the argument?

Judith

Absolutely, I mean Zeiss and ASML are one company each, right? Each company is already serving a very broad, you know, kind of market and you know, very broad demand base that already includes the US. And, you know, we did show in the scenario that, you know, any leverage on Zeiss, if it doesn’t really come with, you know, kind of a much more structural middle power alliance is one that could prove, you know, more feeble than powerful, unfortunately. So, you know, compute is something that we could be building out today. It is something that is already in high demand. It’s gonna be in even higher demand. We’re still early on this trajectory. Yes, absolutely. I would say that’s the right summary of the way that I think about it.

AI safety and security

Henry

Awesome. So I wanted to ask just quickly, because I know we’re approaching getting close to the end of the program. I wanted to ask a little bit about safety. This is one area where I think Europe has performed quite well. And obviously it’s easy to be safe when you’re not building much, but I mean more specifically in terms of institutions like the UK’s AI Security Institute, which is, you know, widely recognised as a world leader, often given first access to models and so forth. And I know there’s an AI security institute now opening in Germany as well. So I’m curious, I guess that I’m gonna try and squeeze in two questions for the price of one. Firstly, how you see some of the kind of pro-growth, pro-acceleration, or at least on the compute build-out side, recommendations of the report.

How you see that as aligning with safety and security in AI, whether you think there is a tension there. And also I’d just love to hear if you have any aspirations or worries or hopes for the German AI Security Institute.

Judith

So hopes for the German AI Security Institute, you know, do what the Brits did. They’re doing a really good job. Honestly, don’t try to reinvent the wheel. I’ve heard different voices. I mean, I know that there are efforts going on, they’re also scouting talent at the moment. I’ve heard, you know, cheering voices, I’ve heard very critical voices that were kind of saying things were going maybe in a slightly wrong direction. Let’s not reinvent the wheel. Look at what AISI has done in the US, in the UK, copy that. Job done. Please, you know, listen to a couple of like, you know, key experts, and let’s just call them and try and replicate what is working really beautifully. I think it’s a really big role model. Now, having said this, we do take safety and security really, really seriously.

And this is one of the big discussions that we had because obviously our timelines are different, they’re slower than the ones in AI 2027. I think all of us have a lot of respect for the work of Dean, Eli and Daniel and so on and so forth. We didn’t want to kind of counteract that or you know kind of criticize their timelines by saying, you know, you’re being too cautious or too aggressive on your timelines or anything. What we, however, didn’t want to do, we didn’t feel like Europe at this point in time would benefit from more safety conversations. We felt like we really needed to make the more industrial case for why this is important.

And also the compute build-out piece that we focus on is not necessarily a compute build-out piece for training because, again, the training market in Europe is unfortunately, at least when it comes to the really, really big kind of training demands, which would come with the types of efforts of frontier labs, we just don’t have a huge market there. And the overall regulatory landscape is not one that really lends itself to like aggressively and like competitively training foundation models in Europe. We think the much more important piece is the inference piece. And I don’t think the inference piece is accelerationist by default, it just means that whatever is there and can be used is accessible. We can actually serve it. So I think in many ways it’s a piece that to me is an equitable piece. And if we have this type of inference—again, I think about it like the grid.

You know, I live in East Germany, which used to be the GDR part, Soviet part of Germany. Well, you know, kind of like, you know, I always say like the fibre-access cable build-out is something that, you know, kind of East Germany has engaged in over the last couple of years because we just didn’t have a lot of that great connection because it was like a Soviet country. And so that’s how I think about compute. Like, guys, I really think like in you know, ten years from now we’re gonna think about this like we think about the grid, and you know, it’s gonna be shocking that a community doesn’t have access to it, like how horrible. Let’s make sure we get them connected to the grid. That’s, I think, much more the analogy that I see in my head.

So I don’t think about this as some type of like luxury, crazy build-out. I really think about this as like the infrastructure build-out that is gonna power, you know, more equitable lives for European citizens compared to whatever global elites are gonna be doing with all the compute that they have and all the things that are gonna be unlocking for them. But back to the safety and security point, we do take safety and security really seriously. We’re big endorsers also of the work of AI 2027. We also think, you know, kind of the general-purpose security provisions in the AI Act are, you know, something that we fully support. We think that, you know, by and large, you know, safety and security always comes first.

We actually think that, because Europe doesn’t really engage on the training frontier and is therefore not meaningfully pushing the needle in a more adverse direction, Europe should have greater weight at the table, including through early access to Fable- or Mythos-level capabilities, to which no European country or company initially had access. Later, obviously, the UK AI Security Institute was given access, but before then none of us were involved. We actually think that we again have the right principles and the right values that we stand for. But the only way to actually allow these principles to have weight in the world is to be taken seriously. And we’re just no longer being taken seriously. We’re not part of the most meaningful conversations.

So that’s why the piece really focuses on let’s make sure we have a seat at the table again, such that the kind of focus on security and safety that Europe has represented for years can actually carry more weight in the overall conversation. And we’re even maybe in a position where in this race between China and the US, we can act as a mediating force, right? And we can, if in doubt, be the voice of caution or be the voice of, you know, kind of calming things down. But to do any of that, we need to be taken seriously again.

Henry

So I know we’re coming to the end of the programme, but Dan, would you like to have the last question?

What would success look like?

Dan

Okay, here’s a final question. So the report brilliantly paints this very bleak scenario. What happens if Europe doesn’t get its act together over the next half decade? Maybe for our final question, could you paint a scenario in terms of what might happen if Europe does get its act together over the next half decade?

Judith

That is a superbly amazing question that I can’t fully answer because I’m currently still engaging in the deep thinking work together with my co-authors of what exactly that would mean. And I think in many ways it is the conundrum that all of us are stuck in right now. What is that beautiful quote? Sorry, I’m not getting it correctly: the future can’t be born yet, but the past is—anyway, scrap that; you know what I mean?

Henry

There’s the Gramsci quote, right? Well, I don’t know if it’s the one you’re thinking of, but you know, the old world is dying and the new one struggles to be born. Now is the time of monsters. I’m not sure if that’s the one.

Judith

Yeah, exactly. That’s the one. That’s exactly the one. And so right now we’re stuck in that. Now is the time of monsters. Here’s what I’ve noticed: the less concrete the benefits that you spell out for people so that they can actually form a tangible emotional connection to that benefit, the more there’s a diffuse haze that tips people into fear about what’s happening. And so it feels like societally that’s exactly what we’re stuck in. We don’t have enough of that. And I’m not talking about the kind of you know bullshit marketing campaigns—however brilliant—that big AI labs are putting out there. I love a lot of that stuff, but it’s not reaching my mother in terms of like she meaningfully understands and feels connected to how AI can meaningfully improve her life. That piece we’re still missing.

And we’re acutely, as the authors of Europe 2031, aware of that, which is why we know how big of a challenge it is, which is why I’m not just out here, you know, dishing out hot takes about how the world’s gonna be better. Because I don’t wanna just be another like marketing lady, who’s saying things that, you know, in the end don’t really have weight or don’t get taken seriously. Because I think it’s not just about positive storytelling, it’s about truly telling a transformative story of how this can go well. This is work that we want to be engaging in. This is work that we want to be doing as kind of the next chapter of Europe 2031. We don’t know exactly what it’s gonna look like, if it’s gonna be another narrative scenario, if it’s maybe gonna be, you know, a different multimedia format.

But, you know, we’ve heard this question so many times, people are really starved for it. And we want to be engaging in the kind of work that allows us to tell that story. What I can say, and so far this is the best analogy that I found, but it really is this, you know, just picturing like electricity and how little we think of electricity. Let me ask you one thing. Henry, you’ve got a nice little lamp and like a candle in the background there and, Dan, I can’t even see the types of light fixtures that you have on the ceiling. But like how many of us think about light nowadays? I tend to think about light a little bit because I have an interior-design foible. But like other than that, I don’t really consider light because it’s just moved into the background.

It’s like so abundantly available to all of us. In fact, noticing the light, like imagine you’re meeting someone for the first time and you’re like, that’s a nice lamp, but like that’s nice lighting. Like they would look at you like you’re a little bit strange. Like, why would you say that? Why would you notice the light? That’s such a strange thing to say. I think intelligence is gonna be a little bit like that. Like it’s going to be everywhere in ways that are gonna make our life so much better, the same way that actual electricity made our life so much better, but it’s gonna be very, very difficult because it’s gonna take different form factors, it’s gonna, you know, take shapes that we hadn’t even considered, but it will make our lives this much better.

I think it’s going to be something like that, where you know, even talking about intelligence at some point will be a bit odd because why would you even notice? Intelligence is everywhere. And yet today we have so much fear because a lot of our life and our value has been built on the fact that you’re smart and you’re hard-working and like these are all the great achievements that you’ve had in life, and we’ve kind of had to unlearn that we’re valuable just because we’re valuable and that we matter just because we matter, irrespective of like the raw IQ output of like what we’re you know, kind of producing in our lives. And I’m not trying to say that you know thinking as a human is not valuable.

I hope it’s gonna remain valuable in the same way that we enjoy going on walks and we enjoy going to the gym and you know, kind of exercising because it feels great. And like I love using my brain, it’s absolutely beautiful, it’s an incredible experience. I much prefer using my brain over running a marathon, I will say that. And I hope that you know humanity never you know loses the joy of engaging in that type of activity at the same time that we’re gonna build structures that will allow intelligence to fade into the background such that other things can move into the foreground, which are gonna be the things that we’re gonna be obsessing with and noticing. Anyway, that’s just my very personal, very hazy kind of way of thinking about this, which is not very satisfying.

But I want to do more work in terms of thinking about this in much more depth and detail. And then we’ll be telling you as soon as we’re ready to share a much more concrete story about how kind of an abundant-intelligence explosion and AI transformation can go well for everyone.

Henry

Amazing. You know, it’s funny, I really like your example of light. It’s something that I do think about a lot because there’s interestingly, I think, a generational difference that I notice with my parents. My parents are obsessed with turning lights off, which you know makes sense for them because you know incandescent bulbs used to blow all the time, they used to be expensive. And I do sort of explain to them: you know, with LED bulbs, right, the actual marginal cost of electricity is zero basically at this point.

Judith

You got it. Yeah, which is crazy.

Henry

Which is crazy, but I mean it shows how quickly I think you can have these generational shifts in like what is for one generation a scarce resource becomes commoditized to an extent. So maybe as you say, we’ll see the same thing with intelligence. Judith, it’s been incredible having you on the show. So I very much encourage everyone, all our listeners who haven’t read Europe 2031 yet to go to Europe2031.ai and read the report. I recommend probably not reading it on a train platform like I did in the rain. Find somewhere comfy, get tucked in.

Judith

A hundred per cent. A hundred per cent.

Henry

And also Judith, you blog at dadalogue.substack.com. Is there any other work that you’d like to steer people towards?

Judith

I’ve written a lot on societal transformation, life post-work, creativity and AI, what is the value of progress? I’ve got a bunch of articles. I’m always digesting in public. They hold more questions than they hold answers, but it’s my best effort at trying to make sense of this wondrous moment in time that we’re finding ourselves in.

Henry

Fantastic. Well, it’s been an absolute pleasure having you on the show. It would be great to have you back. If we don’t get the chance before 2031, we’ve obviously got to get you back in 2031 to assess how accurate things were, but hopefully before then as well.

Judith

Absolutely. All the falsifications that I’ll have to deal with, even though it was never meant to be a prediction. Thank you, guys. Thank you. This was great. Thank you so much.

Henry

Brilliant. All right. Thank you so much.

Robin Hanson writes:

“…while we usually give lip service to the idea that we are open to letting anyone persuade us on anything with a good argument, by the time folks get to be my age they know that such openings are in fact highly constrained. For example, early on in my relation with my wife she declared that as I was better at arguing, key decisions were just not going to be made on the basis of better arguments.”

It’s a funny anecdote. It also contains the seeds of a deep insight about power, rationality, and trust.

In some sense, the insight is simple. When the ability to construct persuasive arguments is unevenly distributed, an agreement to settle decisions by argument can end up ceding power to those who are better at arguing. This wouldn’t matter if argumentative abilities were used only to identify “good” decisions, but they can also be deployed to rationalise biased, self-serving, and flawed ones. Given this, those who are worse at arguing face a dilemma. When they can’t convincingly refute an argument whose conclusion they dislike, should they defer? Or should they simply refuse to play the argument game altogether?

In the context of a marriage, this logic is fairly simple and commonsensical, but one can generalise the basic lesson to other contexts where it becomes more interesting.

Rational Persuasion and Ideological Power

Roughly, one can identify two general factors that determine the stock of persuasive information circulating in a society at a given time.

The first is reality. More persuasive claims and arguments typically better track evidence, survive objections, explain what we observe, and help us predict or control the world around us.

The second is the traits of those who wield the power to produce and disseminate this persuasive information: their motives, interests, biases, allegiances, and incentives.

In modern societies, this “ideological power” is mostly concentrated among an intelligentsia of highly educated professionals who staff universities, media outlets, think tanks, government agencies, NGOs, cultural institutions, research centres, and so on.

This creates a challenge for many people who dislike or disagree with the arguments and recommendations advanced by this intellectual class. If they can’t convincingly rebut those arguments, should they defer, or should they simply refuse to let political matters be settled by those with the strongest arguments?

My thesis in this essay is that this challenge helps to illuminate important features of our modern political situation, especially when it comes to populism and populist hostility to our established institutions. I will also argue that many intellectuals think about this issue in a simplistic and self-serving way.

Highbrow Misinformation

To approach this topic, it helps to begin with something I have written about before, drawing on a term introduced by Joseph Heath: highbrow misinformation.

When people think of misinformation, they often imagine stupid falsehoods, fabrications, and distortions. The Pope endorses Donald Trump for president. Immigrants are eating cats and dogs. Vaccines contain microchips.

This misinformation has distinctive features. It involves unambiguous factual misrepresentations. It originates outside our most prestigious knowledge-producing institutions (science, academia, legacy media). And it is highly legible to the cognitive elites—the academics, journalists, fact-checkers, etc.—within these institutions. (Some academic research literally defines misinformation at the source level: if it appears in the New York Times, it’s reliable information; if it appears in Breitbart, it’s misinformation.)

So understood, misinformation is something “they” produce: populists, foreign disinformation campaigns, grifters, conspiracy theorists, cranks, and alternative media. This model of misinformation, therefore, misses subtler forms of misleading communication that originate and circulate within our prestigious institutions.

Consider claims that are true but predictably create false impressions.

For example, one might argue that the world will likely be much poorer as a consequence of climate change (true), without mentioning that this forecast is relative to a counterfactual world without climate change, not to the present.

Or one might respond to claims that certain immigrant or minority groups are over-represented in certain crimes by pointing out that most of those crimes are committed by the country’s white majority, which might be true but might also be perfectly consistent with the claim it is deployed to rebut.

Read more

His new book, The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning, combines an excellent, highly accessible, up-to-date introduction to modern AI with original, provocative arguments about what the technology means, both politically and metaphysically. Most commentators sit somewhere on a line running from “AI is overhyped nonsense” to “AI is radically transformative.” Bob goes much further: he thinks rapid advances in AI confront our species with something like a moral test of the kind a God might set, which we can only pass by overcoming our evolved tribal biases and becoming a cohesive global community.

We had some questions! Among many other topics, we discuss:

• What Wright means by a “God test”

• Whether AI is a god test on the scale of nuclear weapons, and why the coordination and verification problem is harder this time

• How large language models (LLMs) come to represent meaning, and the case for modern AI as “reverse engineering” the mind

• The sceptic’s case against imminent transformative AI: jaggedness, sample inefficiency, disembodiment and continual learning, and why I (Dan) hold longer timelines than Wright

• Evolutionary psychology versus the blank slate: why Wright expects jagged rather than general intelligence, with a detour through Richard Sutton, Skinner and the “bitter lesson”

• Technological determinism and agency: why true agency may be impossible inside competitive environments

• The default trajectory: destabilisation, authoritarianism, and why a breakneck race to superintelligence could bring authoritarianism to America “through the back door”

• The US and China race: steelmanning Dario Amodei’s arguments, the Superintelligence Strategy paper, and Wright’s contrarian (in America, at least) reading of Chinese intentions

• The security dilemma and threat inflation: why defensively motivated moves get read as aggression, from the First World War to the South China Sea

• Negativity bias and the discourse: my challenge that commentary on AI, and the book itself, tilts too far towards negativity, alarmism, and catastrophism

• Does evolution have a purpose? Smolin’s cosmological natural selection and William Hamilton on directionality

• Consciousness and epiphenomenalism

Links

• Robert Wright’s Nonzero (newsletter and podcast)

• The God Test (Simon & Schuster)

Mentioned in the episode:

• “AI as Normal Technology”, Arvind Narayanan and Sayash Kapoor

• Allan Dafoe on the 80,000 Hours podcast, on why technology is unstoppable and how to shape AI development anyway

• “Machines of Loving Grace”, Dario Amodei

• “Superintelligence Strategy”, Dan Hendrycks, Eric Schmidt and Alexandr Wang

Transcript

• Please note that this transcript is lightly AI-generated and may contain minor mistakes.

Dan: Okay, welcome back. I’m Dan Williams, back with my co-host, Henry Shevlin, and today we’re joined by a true legend, the great Robert Wright. Bob is the author of the Nonzero newsletter, host of the Nonzero podcast, and author of some genuine modern classics, including some of my personal favourites, like The Moral Animal, which is one of the first and, I still think, one of the best introductions to evolutionary psychology, as well as Nonzero, which is a kind of sweeping story about the biological and cultural evolution of cooperation and complexity. More recently, he’s the author of The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning, which is going to be the topic of our conversation today. So Bob, welcome to the podcast.

Bob: Thanks for having me.

The God test and the cosmic frame

Dan: In terms of a continuum of views about the significance of AI, at one end you’ve got the people who say it’s all just a con, it’s hype, it doesn’t really work. Then you’ve got the people who say it’s an important technology, but fundamentally just another technology. Then you’ve got the people who say, no, this is radically transformative, we’re building potentially superintelligent agents that are going to completely upend the social order. And then we’ve got you, saying even more significantly than that, this is a God test and the grounds for a cosmic reckoning. So firstly, what do you mean by that? And secondly, why should we think it’s such a big deal?

Bob: I think it’s going to be, first of all, a total earthquake. A force of great magnitude is being unleashed. In that sense I’m on the same page with some other people, including, I would say, both doomers and some accelerationists who, for reasons I don’t understand, think that although we’re going to enter a singularity beyond which nothing is clear, everything’s going to work out fine. That would make more sense to me if they said they were religious and had religious faith in it. Otherwise I do not understand the basis for the optimism. It’s true that in a sense I step back further than most people in both camps in terms of how cosmically I locate our situation.

On the one hand, in the book, the thing I’m probably proudest of is trying to make clear, especially to a non-technical audience, what the secret sauce of the deep learning revolution is: why the capabilities of AI have grown so fast and will keep growing, and what is going on inside a large language model, to the extent that we can tell. So there are some very mundane aspirations. But it’s true that I make a show early on of saying I’m going to step back and view this in very cosmic terms, in the context of the whole three-and-a-half-billion-year history of life. And I do think, and some others would agree with this part, that if you ask Eliezer Yudkowsky whether the birth of silicon intelligence on this planet deserves an asterisk, even in the context of all of life, an asterisk that very few others rank with, he’d say yes. So I’m not uniquely cosmic in the way I look at this.

Now, when you start talking about what I mean by the God test, I should say the most important meaning of that is metaphorical. I’m always happy to talk about the possibility that some larger purpose is unfolding on the planet, and I do to some extent, largely in the appendix. But the main meaning of the term is that I think artificial intelligence poses us as a species with the kind of test that God would give, the kind God gives in the Bible: salvation is possible, but you’ve got to get your act together. You see things like that in the Old Testament and the New Testament, and in other religions. It’s a moral test. I think our species needs what you could call a moral upgrade, because I don’t think we can navigate the AI revolution successfully except as a cohesive global community, and we are a long way from being that. In order to become that, we’re going to have to become in some sense a better species morally, and I mean something a little less ambitious by that than it sounds. It’s not entirely impossible that we could succeed. But that gives you some sense of why the word cosmic is in the subtitle.

Is this like nuclear weapons?

Henry: Just a very quick clarification. I’m reminded of nuclear weapons here as another possible God test. We had this period where suddenly we had the capability to at least destroy civilisation, if not our species. Do you think it’s fair to call nuclear weapons a God test we’ve already, to some extent, passed? Or is this a different order of magnitude of test?

Bob: I think it’s quite different. There are parallels. One thing I’ve heard people say about AI is, hey, we can do this thing with China, because with nuclear weapons we were adversaries, if not enemies, with the Soviet Union, but we worked out some arms control deals. That’s true. But I think the degree of international coordination, even international governance, that’s going to be required this time, and the nature of it, mean you’re just going to have to be on better terms with the other countries than we were with the Soviet Union. This is not a few isolated arms control agreements. First of all, as people have noted, with AI you’re generally talking about a bigger verification challenge. The degree of monitoring, if you want to monitor big training runs, they’re pretty conspicuous, so it’s kind of doable. But even now, some people are starting to say, and Anthropic hinted at it in a paper they wrote about recursive self-improvement, that maybe at some point we should talk about a pause, or a slowdown, and if so, that will need to be globally coordinated. That’s fine, but with the current politics of the United States it’s hard for me to imagine that happening unless our relationship with China changes, because there will just be too many people saying, they say they’re pausing. We’ve already seen how, even though we shut off their supply of great microchips, they’re doing pretty incredible stuff. I guess they’re doing it with something a little less conspicuous than Elon’s Colossus Two in Tennessee. So I could go on, but ultimately it goes beyond a couple of specific quasi arms control agreements, and even those are in some cases more challenging to enforce than a nuclear treaty.

Why so bullish on the current paradigm?

Dan: I really want to get into the geopolitics and the arguments about international coordination, and also the cosmic framing. But we’re going to have some people listening or watching who are more sceptical of the technology itself. It would be helpful, Bob, if you could say a little about why you think, not just in principle we could build AI systems that substitute for human labour across the board and reach something like AGI and maybe artificial superintelligence, but why you’re so bullish on the current paradigm.

Bob: Okay. I gather you have two kinds of audience, audio and video. This won’t surprise the video audience, but maybe it’ll surprise the audio audience: I was a journalist in 1983. I’m pretty old. I interviewed Geoffrey Hinton for a piece I was writing on AI. Since then I’ve intermittently written about technology, not exclusively, but I’ve kept up with it. After the large language model got my attention, and it was the first time AI really got my attention in a big way, I went back and read the piece I’d written. I’d written the Time magazine cover story when the IBM computer beat Garry Kasparov for the world chess championship, and honestly I thought that was such a boring story that I steered it into a story about consciousness. I just didn’t think it was that interesting what the machine was doing, because it wasn’t very much like what a human mind does.

But back to Hinton. When GPT-3.5 and then 4 got my attention, I went back and read that piece, which did talk about his maverick approach. The term neural networks is in the piece, a favourite phrase of his at the time, along with massive parallelism. But I realised I really did not understand the potential of the paradigm. I didn’t understand the paradigm. My example of a neural network was kind of legit, because it came from a guy who had co-authored a paper with Hinton, but he was a psychologist who meant something different by neural network than what was most important to Hinton. The neural network I described was in theory going to be equipped with the ability to process language. But with that kind of network, where each node was going to represent a certain sense of a word, humans would have to implant in the machine the human understanding of the connection between the meaning and the words. You’d have to somehow, maybe you could automate it by feeding a dictionary into it, but somehow the explicit connection between individual words, or their specific senses, and meaning would have to be engineered. Humans would have to come up with a way of representing meaning and put it in the machine.

When I went back in 2023 and looked at a lecture Hinton had given in 2018, and started looking at how modern neural networks actually work, I realised we don’t have to put the meaning in. You just make them better at a task. It could be next word prediction, fill in the blank, translation into a foreign language. But all of these tasks, from the machine’s point of view, are just: here’s a bunch of gibberish, figure out the next gibberish. We’re not telling it that the symbols mean this or that, and yet, because we give it a broad framework for encoding each word at the beginning, we say, look, each word, and this is a slight oversimplification, should be a bunch of numbers separated by commas, whatever numbers you want, just keep changing the numbers until you finally get good at the task. So there’s no point where we’re telling the machine what the meaning of the words is, and yet we now understand that the machines wind up using the numbers as a means of representing the meaning of words.

Another slight oversimplification: imagine a graph with two axes. You’re plotting, say, animals, with velocity on one axis and lethality on the other. Tigers would be high on both, rattlesnakes high on one and not so high on the other. That’s two dimensions. In effect these machines, in an implicit way I won’t get into, are using a bunch of numbers to represent the meaning of words in a high-dimensional space. And it turns out, as you might expect, that words close in meaning are close to one another in that space. So there is this mapping onto meaning, and there’s actually a second kind that gets done with multimodal.

That was a mind-blower for me. This would save you some time if you didn’t really have to say much about the meaning of the words to the computer, or how it should represent that meaning, aside from saying it has to have some connection to these numbers. And then I realised you can apply this everywhere. It’s the same way Elon’s cars learn to drive: you give it human-like input data and then the output data that humans would provide. With Elon, the input data is visual and the output data is turn the steering wheel slightly to the right. And you can get the machine to, I think a fair way to put it, though not everyone in the field would agree, reverse engineer the functional components of the human mind. They don’t work exactly the way those components work, but the functionality, such as the representation of meaning, or in the case of image recognition something as fine-grained as edge detection neurons, gets reverse engineered. These machines, I’d say, reverse engineer an edge detection mechanism comparable to edge detection neurons.

I honestly think, even in the AI community, one reason I’m proud of some of the exposition in the book is that I’ve spent my whole life as a journalist trying to make things accessible to a lay audience, and experts are just inherently not good at this. It goes with the territory: when you know a lot about a field, you lose track of what the average person doesn’t know. If I ask why more people in the field don’t say, look, these things can reverse engineer any part of the human cognitive or perceptual apparatus, and all we have to do is feed in the data, why don’t they put it that simply, I think in part there are some disagreements, some people would quibble with my saying that, but in part it’s just that they’re experts, and there’s a lot they take for granted that would help people understand the magnitude of what’s unfolding, and they just don’t put it that way.

Semantics, syntax, and “normal technology”

Henry: The description of semantics you’re giving reminds me of an expression by the philosopher John Haugeland, who said, contrary to the view that you need to program in the meaning of every symbol, the symbol grounding view, if you take care of the syntax, the semantics takes care of itself. He was saying that back in the 1980s, and I think that captures what we’ve seen: that models, given enough data...

Bob: Well, that was early. Was he talking about neural networks?

Henry: He was expressing the hope that we wouldn’t need to do this deep, elaborate symbol grounding for every symbol in a system. So he was offering an optimistic picture of how things would work out. But just to press you a little further. Dan and I are completely aligned on the incredible linguistic capabilities of LLMs, and to be honest we’re probably aligned about the broadly transformative potential of AI too. But we’re also both fans of this “AI as normal technology” view that we’ve discussed on the show a few times, which says: yes, AI is a huge deal, it’s going to be one of the biggest technologies in human history, but it’s not unprecedented. It’s up there with the internet, or electrification, or maybe the industrial revolution. Your view is that it’s more than that. So what makes it different from these other big technological paradigm shifts?

Bob: First I’d say it’s hard to separate some of these things. You can’t imagine modern AI without the internet, because it trains on the internet. But that said, the advent of a new kind of intelligence that is not organically based and is as smart as the smartest organically based intelligence, that’s a first. Life has been here three and a half billion years. That’s a first. Secondly, to look at jobs, for example, there’s never been a technology where it was so at least plausible, we don’t know yet, but plausible, that this time, when it takes the jobs, there won’t be other jobs. It’s an open question. I can imagine scenarios, but it’s never been such a good question whether there will be anything for us to do.

And if you go down the line of other areas where it will have a huge impact: jobs is one, interpersonal relations is another, friendships, romance. Parents are going to be freaking out more and more about why their kid is spending so much time with the machine and less time with human friends. I’m not saying we won’t get through any of this, or all of it, eventually. I’m just saying the combined impact will be unprecedented, especially on such a short time scale. To some extent that’s just a function of how things happen these days. In my book Nonzero I spent a lot of time on the printing press, which was in many ways analogous to, well, narrowcasting, the internet, social media. But that took a long time. It took decades before you had actual wars plausibly attributable to the printing press. This is going to happen very fast, and along many dimensions. I think it’s going to be geopolitically destabilising, politically destabilising. We could talk about the persuasive capabilities of these things. When you add it all up, the net impact is going to exceed, certainly on this time scale, anything in history. That’s why I’d like to see us slow down and proceed more deliberately, and as soon as possible proceed as a global community, and recognise that this is a challenge we can only respond to adroitly as a global community. I’d like to hear more about your side of the argument. You can define normal in a way that would make this work; if you define normal as abnormal, I’m on board. But what’s the definition you’re using?

Henry: I’m going to pass to Dan, because in our debates on this I’ve been more on the transformative side and Dan’s been more on the normal technology side. Right, Dan?

Dan: To an extent. The way I’d put it is, once we’ve got AIs that are smarter than human beings across the board, then yes, I agree with you, Bob, that’s not a normal technology. That’s, to put it mildly, a unique, sui generis technology. But I still think there’s a sceptical position here that says, why should we think we’re on track over the next several years to get to that point with the current AI paradigm? You do a really great job in the book showing that there’s a really important sense in which LLMs genuinely understand meaning and are really intelligent in lots of ways. But a sceptic is going to say, okay, but there are still so many things they can’t do. They can’t do continual learning the way non-human animals and human beings can. They’re extremely sample inefficient, they need an enormous amount of data to learn various capabilities. They’re disembodied. They’ve got this jagged competence profile where they’re superhuman at some things and really kind of stupid at others. So a lot of what you’re arguing in the book is not just that the current paradigm is really impressive and incredibly intelligent along specific dimensions, it’s that we are on the cusp, over the next several years, of reaching truly transformative AI. I take that seriously as a possibility, but I don’t think it’s the most likely scenario. I think I’ve got longer timelines than you do, Bob. So what’s the case for thinking this is likely to happen over the next five or ten years, rather than more gradually over the next several decades?

Bob: Well, first, you’ve pointed to something they’re not good at, which is real-time, continual learning, the way a worker learns on the job, the way we all learn things through our lives. I’d also say that a few years ago you could have pointed to some things people were laughing at AIs for, and they’re laughing less now. Reasoning is one, and that’s pretty big. You can still find it making some stupid reasoning errors, but the whole chain-of-thought reasoning thing was a true and very important innovation that just wasn’t there three years ago. That’s super powerful. And relatedly, training techniques have got refined in what’s called post-training. By the way, the remarkable thing about this vocabulary, and maybe they’ve changed it, but as of a year or two ago, there was such a thing as pre-training and such a thing as post-training, but no training. And I’m thinking, wait, the people who came up with this are our leading lights? We’re trusting them to guide us through this revolution? I just want to put a wall around Silicon Valley. You want to build machines, fine, but somebody else should be in charge of making sense of this, if that’s your approach to coherent terminology. I’m joking. But there have been real refinements in training that have added real dimensions of capability.

I want to get back to the jaggedness. You said it’s disembodied. Well, yes, but again, the basic approach to reverse engineering parts of the mind, and I will defend that metaphor, can be applied to any sensory channel: tactile, olfactory, visual. And there is real progress in robotics. Some things will take longer than others, but I don’t see a particular place where there’s a super fundamental obstacle ahead. Now, as for continual learning and, relatedly, the jaggedness, this gets back to one other thing I say in the book. I said some people in AI might not sign on to reverse engineering as a metaphor. Here’s something more of them might not sign on to, though I think most of them probably would, except maybe Richard Sutton, the coiner of the famous “bitter lesson” term and paper. It’s been most common to refer to pre-training and post-training as learning. I think that’s fair, but for some purposes it’s really more analogous to biological evolution. Language is a good example. During training, yes, the machines learn how to speak English, kind of, and that’s something a child learns, so that’s learning. On the other hand, they also develop this system for representing meaning, and, in the case of image recognition, these edge detection filters. Those, I’d argue, and I think it’s completely safe to say in the case of the edge detectors, and pretty safe to say in the case of a system for representing meaning, are products of biological evolution in humans.

The reason I bring this up is that there still prevails, in parts of the AI community, and, having listened to that pretty famous Dwarkesh interview with Richard Sutton, I think he’s an example of this, the idea that the human mind is a blank slate. That it’s like B. F. Skinner. I wish Dwarkesh had just asked Sutton, so you’re a hardcore Skinnerian? Because that’s the only way I could make sense of what he was saying. And I’d love to look further into the origin of the term artificial general intelligence. When I first heard it back in 2017, I thought, wait, do they think intelligence is just this general thing? Because, as I said, I’ve written about evolutionary psychology, and in that field the view pretty much is that evolution just kept improvising and building different things. So the mind is a jumble of functional things that, yes, are highly integrated, but it’s still not a general purpose intelligence machine. It’s good at threat detection, it’s good at sensing the emotional vibes of people, it’s good at making out visual objects, but these things evolved at different times, so there’s different equipment.

And I don’t see why we should expect the evolution of AI to be any different. So jaggedness is what I’d expect, because I don’t think there is a general blank-slatey thing. I was reading, or listening to, this biography of Demis Hassabis, and at some point he says, if we can solve intelligence, and I thought, that’s a weird phrase. I don’t think of intelligence as this one problem that you’ll at some point get the solution to. Maybe it’s out there, but I don’t think that’s what the human mind is. Look, who am I? With all due respect, the guy’s a genius, I’m not, and I don’t want to sound like too much of a jerk. But the answer to your question, if there are still these big gaping gaps in functionality, what’s your basis for hope, is: well, that’s the way evolution works. It’s in the nature of intelligence that it’s not just this general blank-slatey thing. That’s my view. Now, continual learning is a specific problem. First of all, if you’re talking about the practical impact, like can it replace all workers, it’s not hard for me to imagine simple shortcuts that should worry workers who want to hang on to their jobs. When Mark Zuckerberg starts monitoring the keystrokes of his workers, that’s one. You just update your LLM, or whatever, with fine-tuning, with the input derived from that monitoring. And as a practical matter you’ve solved the continual learning problem. You update it every once in a while. As a practical matter, that does not seem to me like it’s going to get in the way of the huge impact I’m talking about.

Inevitability, agency, and competitive environments

Henry: Can I ask about the attitude of inevitability that maybe I’m incorrectly reading into a lot of what you’re saying? It seems like there’s this path you see humanity as basically locked into now. I’m curious, firstly, whether I’m understanding that correctly, and secondly, how you’d respond to someone who says, look, sure, with evolution you’re dealing with certain deep grooves in some latent biological space. We’ve seen convergent evolution towards intelligence, multicellularity, social organisms. But technology’s different. We have more optionality with technology. Things could have been invented in a different order; if this country had gone to war with that country, things could have worked out very differently. So, do you see yourself as more on the technological determinist side, at least with regard to AI? And how do you respond to someone who says biology and technology are different, there’s more optionality in tech?

Bob: There is, in principle. In both cases, what we’ve seen so far, as a rule, is that what’s highly probable, if not inevitable, is the evolution of certain capabilities and properties. In evolution, it wasn’t inevitable, I think, that the first form of intelligence at our level would look like us, have five fingers. But the property was probably pretty likely all along. Certain things are independently developed many times in evolution: multicellularity, vision, flight. Those properties were likely to be discovered. I think you have the same thing in technology. But you’re right that in principle, because we are the environment of technology’s evolution, and unlike the environment of biological evolution, we’re aware that we’re playing that role, in principle we could change things.

There’s a guy at Google I still want to track down and invite on the podcast. He was on Rob Wiblin’s 80,000 Hours podcast a couple of years ago. He’d done what sounded like a PhD dissertation, and it spent a lot of time on the concept of agency: when can people exert true choice, true agency? As I remember the take-home lesson, it was when they are not in a competitive environment. If you’re a company trying to maximise stock value, you don’t really have much agency. Right now, if you’re Google or OpenAI or Anthropic, that’s why we have all these guys hinting, yeah, it might be a good thing for the world if we slowed down, but I’m afraid I can’t do that, because they’re in a competitive environment. That’s why I put so much emphasis on this, and I wish I’d put it in the book, I wish I’d tracked down this quote, because it’s a point of profound philosophical significance: true agency just cannot happen in certain kinds of competitive environments. And in the current international environment, humankind does not have agency over this technology. That’s the way to put it. So yes, in principle we do have agency in steering technology, and occasionally it’s been exercised, especially when it can be exercised at a national level. But sometimes it’s been exercised internationally, like ozone depletion, not to be confused with climate change. They actually handled that. It was a modest technological adjustment, more or less getting rid of aerosols. But it happened, and it happened internationally. So it’s possible.

Dan: Just one quick thing, Henry, then I’ll bring you in. I think the person you’re referring to is Allan Dafoe, on technological...

Henry: Yes, that’s what I was going to say. Allan.

Bob: Yeah, I think you’re right. God bless that man.

Dan: We’ll put a link to Allan Dafoe in the show notes. I think that’s a nice transition onto these questions about the idea of a pause.

The default trajectory and authoritarianism

Dan: As I understand it, a central argument in the book is that the default trajectory here is extremely worrying. Unless we get our act together, unless we achieve a certain kind of international cooperation and coordination, and we’re only going to do that if we overcome tribal cognitive biases, things are going to be really, really bad. So even though you’re not quite an AI doomer, because you think we’ve got a chance of intervening to avoid the worst-case scenarios, is it fair to say, Bob, that you think the default trajectory is extremely worrying and most likely really bad? And if that’s a fair characterisation, why do you think that?

Bob: If by default you mean the world failing to achieve an unprecedented degree of international coordination and governance, then I think it’s bad for more than one reason. One is just the magnitude of the earthquake. If this proceeds at its current velocity, completely unconstrained, with only the kind of regulation you can get at the national level in an environment of intense international enmity, which in many cases is not much regulation, then just the upheaval, when you add it all up, the social impact in America alone is going to be very destabilising. And you know what happens in periods of great disorder and chaos: it’s ripe for authoritarianism. As it happens, this technology is great for authoritarians. So this technology is both creating the circumstances for an authoritarian takeover and providing the tools.

This is a good segue to another part of the answer: geopolitical destabilisation. This is why I think Dario Amodei’s plan, unless he’s updated it, to literally get to superintelligence before China and bring China to its knees, and that’s a totally fair reading of what he says in “Machines of Loving Grace”, he doesn’t mention China there, but he says authoritarian bloc, and he’s co-authored things on China with one of the most extreme China hawks in America, Matt Pottinger, his plan is to bring China to its knees and then give it some ultimatum that he’s a little vague about. But my point for now is that getting to that point is so destabilising. The Superintelligence Strategy paper that Dan Hendrycks was lead author on, its most important point was kind of buried: if two nations both buy the premise that superintelligence confers hegemony, and buy the premise that, because of a dynamic of acceleration, which I think is evident, being a couple of months ahead in the race to this threshold could ultimately mean you attain complete hegemony, then obviously the superpower that’s two months behind has a strong incentive to resort to extreme measures. If they’re hearing from the other superpower what Dario Amodei in particular is saying, and what many American politicians are saying, which is we’ve got to win this race, and then America will, and this guy Alex Stamos, a very impressive, I’m sure good, guy, a famous cybersecurity expert, said America has to “dominate the 21st century.” That’s not quite as extreme as what Dario says, but if I’m China and I hear a bunch of Americans saying that, and you’re two months ahead, and it does look like things are accelerating, well, I might just resort to extreme measures. And it might not even stop with the most obvious thing, which is putting an end to the functionality of those factories in Taiwan, from which China is not getting any advanced AI chips anyway. It might go further than that. And you’re talking about two nuclear superpowers here. So you tell me, how much attention have you seen to the issue I just raised in the discourse? Maybe I’m missing something.

Dan: How much attention have I seen? I’ve seen a lot of debate over whether that strategy, which is being pushed by Amodei and others, is the right one.

Bob: But this specific point, that this invites attack from China? Again, it’s in the Superintelligence Strategy paper, I give them credit, I’d have flagged it a little more prominently. You’re starting to hear a little of it. Look, I don’t want to spend too much time complaining about what is and isn’t in the discourse. But this is part of the answer to your question, and I want to close it out by connecting it to the first thing I said. If we do what is basically Dario’s prescription, notwithstanding the occasional Anthropic paper that says, yeah, really, we should think about slowing down at some point, if we really pursue his prescription, which is in fact a breakneck race to superintelligence, because he genuinely believes this, and I think for him it’s not just a corporate talking point designed to fend off regulators, it’s ideological, and I genuinely respect that, he has principles, he has ideological principles, but I think they lead to a prescription that, if we follow it, not only courts the risk I just described but will lead to such rapid destabilisation of American society, which I submit, by virtue of its system of government, is less good at controlling instability than China is, that we may wind up, for reasons I described earlier, with authoritarianism coming to America through the back door. Dario’s whole thing is motivated by his fear of authoritarianism. He thinks China wants to make the whole world authoritarian. I haven’t seen evidence of that, but he thinks it’s the case. Fine. I’m just saying the kind of race he’s talking about has a good chance of bringing authoritarianism to America through the back door.

Steelmanning Amodei, and reading China

Dan: I think that’s a really strong argument, and one of the most compelling parts of the book. But I still think you have to look at the risks and dangers on both sides. You might steelman Amodei’s view by saying, look, of all the scenarios that could play out here, the current one, where frontier companies in the US are racing to superintelligence, all things considered is not that bad. For one reason: it’s happening in a liberal democracy. Saying that fifteen years ago would have been a bit more comfortable than it is now, given the Trump administration and so on, but relative to the political regime in China, these companies do exist in a liberal democracy, where there’s some kind of democratic accountability. And the frontier companies leading this race are demonstrating a surprising amount of concern with safety, genuinely employing AI safety and ethics researchers, putting a lot of funding behind that, being at least somewhat responsible relative to what you might expect from corporations like this. If you compare that scenario to one where, for example, China gets to superintelligence first, then it looks actually quite good. So if you could actually have an internationally coordinated pause, how confident are we that the scenario that would play out as a consequence would be superior to the current one?

Bob: Well, I’m not advocating that China get to superintelligence first. I’m advocating that we start right now having a very serious conversation with the rest of the world about nobody getting to superintelligence first. That said, if you accept the premise that superintelligence is this threshold that confers dominance, and goes beyond military dominance, which I don’t think is crazy, and I’ve never heard this spelled out, but probably somebody has, I assume they’re talking about things like being able to say to your AI, go infiltrate China’s social media, or America’s social media, and instigate a takeover of the government. I think they’re talking almost about that level of power. I don’t think that’s completely crazy. And I grant that if either nation gets that, I find it kind of creepy, honestly. Certainly if China gets it, I don’t feel comfortable that it would be a great world, which is why I’m not advocating it.

But I also think there’s an assumption that right now part of China’s foreign policy is the aspiration to remake the world in its image, that it wants other nations to have its system of government. I think that premise is almost wholly without supporting evidence, because it’s another one of these things that just doesn’t get debated much in the mainstream halls of Western discourse. I think China is, in a way, the opposite: a very pragmatic, realist country. America, obviously, has said it is our aspiration, it’s Dario’s aspiration, to remake the world in our image, we want liberal democracies everywhere, and if you’re not one now, we will sanction you or invade you or bomb you. That is our policy. And, by the way, is a country that just seems to have a habit of invading and bombing other countries the country you want to get to superintelligence first? If you’re an observer from Mars looking down, China hasn’t attacked another country since 1979, and America has done it under every single president it’s had since 1979. Is it so obvious to an objective observer that things will work out fine if America has the superintelligence? Maybe you could say, well, then they won’t have to invade countries. I just think the premises underlying the whole discourse about China are not as carefully examined as I’d like.

But I mainly want to emphasise: I’m not saying let’s let China get to superintelligence first. I’m saying job one is just to see if we can talk about slowing down a little. And granted, that would have to be verifiable. That’s why my book’s proposals in the non-political realm seem to some people hopelessly ambitious: encouraging people, including in America, to get a little better at transcending some of the cognitive biases that undergird the psychology of tribalism, so that it will be less easy for leaders who want to inflate our fear of other countries, and get us to bomb countries like Iran, to do that. I know that’s ambitious, and I’m not saying it’s probably going to work, it probably won’t work, we’re probably toast. But my argument, and the book is an argument, is that we have to, as a species, get a little closer to enlightenment. I’m not just talking about America. In all these countries, all populations are pathetically susceptible to threat inflation, and it’s always in the interest of political leaders, and others like arms makers, to inflate threats. So I want to emphasise what I’m saying: not let’s let China take over the world, but how about we understand that it’s in our mutual interest for nobody to do that, and the first step would be to have rich enough discourse for some degree of slowing down and international coordination to not sound completely crazy.

Henry: There’s a lot I agree with there, but two areas I might push back on. One: yes, China hasn’t had a recent track record of going to war with its neighbours, but it has absolutely acted in quite aggressive ways towards a whole bunch of countries in the West Philippine Sea. It’s engaged in constant low-level aggression with Vietnam, the Philippines. It’s been building artificial islands and reefs. It’s been on a large-scale international enterprise of loans via the Belt and Road Initiative, giving it more and more control over countries around the world. Okay, it’s not bombing Iran, but there are lots of very reasonable, not crazy war-hawk, people who look at the way China’s acting and say this is a country that does have a global agenda that we should be aware of, and that is at least to some extent in tension with that of the United States.

Maybe the second, more fundamental point I’d press you on: the worry here is not necessarily that China’s going to get superintelligence first, but that superintelligence is really one of the very few arrows in America’s quiver where it currently has a significant lead over China. On a whole bunch of other really sensitive military or dual-use technologies, from drones to robotics to shipbuilding to batteries to energy generation, China’s steaming ahead of the West. Superintelligence may be the only really decisive lever the West has if it doesn’t want to end up with a China-dominated century. So what would you respond to that?

Bob: Quickly on the last point before I return to it: I’m not advocating that we have a century dominated by any one country. I’m advocating that we avoid that. On the first point, I assume you’re not saying that the things you mentioned, like the Belt and Road Initiative, throwing its weight around regionally, distinguish it from America. You’re not saying that, right?

Henry: No, I’m not saying that. But equally, there’s a certain tendency to paint China as fundamentally an inward-looking power that only wants to secure its own economic future, and I don’t buy that. China does absolutely have a global agenda that involves increasing its influence in Africa, in Latin America, in absolutely dominating the South China Sea, or West Philippine Sea, pressing land claims and so forth. So it’s not this purely introverted power that some people make it out to be. I’m not suggesting you’re suggesting that.

Bob: On the point that it’s not just concerned about its national security: I had on my podcast Dmitri Alperovitch, who founded CrowdStrike and had written a book, and he’s a China hawk. I said to him, I read your book, and the great thing about it, as China hawk books go, is that when I look at your assessment of China’s motivations and why they want to exert this regional influence, it can all be explained as a means of preserving the trade conduits it depends on. He emphasises that, well, there are certain parts of the ocean that are super shallow, and I forget how it all works out. But the main thing is, and people can check the podcast to make sure I’m not exaggerating, he said, well, yeah, that’s true. And I want to say: the biggest human cognitive contributor to war, related to threat inflation, is the tendency of people on both sides to view things that may be defensively motivated on the other side as offensive. Political scientists have a term for this; the term security dilemma kind of encompasses that. I’d say it’s not exactly that, but it encompasses it as one of its main strands. You see it again and again. You see it in the way Americans think of China, the way China thinks of Americans, the way Iranians and Israelis think about each other. I can go on all day citing chapter and verse. Famously, this may be the reason World War One happened. It’s a positive feedback cycle: they did this thing, that’s a threat to us, so we’re going to do this thing, and then the other side sees that as a threat, and so on. It’s been much commented on.

I just wanted to say that what you said, that China throwing its weight around regionally is a sign of aggressive aspirations that are not purely defensive, not purely in the name of national security, I’d say that’s an example of that dynamic. But before I go on, I can pause while you rebut me.

Henry: I think it gets a bit blurry, and I agree that very often seemingly aggressive measures are motivated by insecurity. That’s basically the history of a large part of the Cold War: the whole idea of containment, worries about domino strategies. So there’s no neat distinction between defensive versus aggressive powers. But the one area I’d say is that, given that the United States and China do constitute very different ideological visions for the future, I don’t think it’s the case that totally peaceful coexistence is the default option. I think they have quite distinct interests in different parts of the world, whether that’s the United States and the Monroe Doctrine in Latin America, which is obviously at odds with China’s vision of having large amounts of influence, economic mainly, but also perhaps political. But I’m not sure that’s a fundamental disagreement, and I do want to hear what you say about my other point, which is that superintelligence is one of the very few assets the United States has if there is going to be this competition.

Bob: Again, my premise is that we have to make sure there isn’t the competition. And it’s related: you’re asking what the chances of peaceful coexistence are. I’m not saying they’re high. I’m saying, and it’s an argument, so it could be wrong, but it’s an argument laid out in the book, that this technology has properties that force us to fundamentally rethink things we’ve always taken for granted, like that the peaceful coexistence of all the world’s nations is impossible. There was a time when a lot of things we take for granted were considered impossible. If you’d said seventy, eighty, ninety years ago, sixty maybe, right after World War Two, that someday France and Germany would have the same currency, the reply would have been, well, which country will have conquered which? You wouldn’t have imagined that. As it turns out, the single currency has had its upsides and downsides. I’m just saying that forms of cooperation and collaboration that seem inconceivable do happen. There was a time when there had never been institutionalised peace among groups of people more than ten miles apart, and now we have huge swathes of peaceful coexistence. There’s Peter Singer’s book The Expanding Circle, which argues that there’s been a kind of moral advance over time that humankind has exhibited. Peter and I have somewhat different ideas about what the main engine of that has been, but I think it’s happened. So, as I understand it, your question is, if we don’t dominate China with superintelligence, how will we dominate China?

Henry: Or rather, the worry is that if the United States does not use superintelligence to at least equalise the power differential with China heading into the 21st century, then China’s massive and growing lead in other strategic technologies, drones, energy production, shipbuilding, will mean the 21st century ends up being one dominated by the ideology of the CCP. That’s the worry.

Bob: Well, most people who talk about superintelligence and assess its strategic significance don’t take it as the kind of thing that could balance China. It’s like, balance their drones? No, if we have superintelligence, their drones won’t matter, we’ll have better drones. It’s a technology of global dominance, according to most people who think about it. So to think of it as something that will just maintain the balance, I don’t, now, I haven’t thought this through, I’m a little agnostic about superintelligence, its properties, its likelihood, its date of advent, but the premises underlying most of the discussion of it, as I understand it, preclude the possibility of one nation having it and it being a balancing thing. No, it’s a dominating thing, in these discourses.

Henry: I’m inclined to agree with that, and then I want to pass back to Dan to move the show on. But I think it illustrates how a lot of one’s commitments about the nature of AI hang together. If you really think AI superintelligence is just a transcendental technology in some sense, then the idea that it could ever serve as a geopolitical balance doesn’t make any sense: whoever controls it determines the future of our light cone, let alone the future of the 21st century. On the other hand, if you have a more “AI as normal technology” view, then superintelligence might start to look like something that could be in column A to counterbalance other industrial capabilities in column B. So it’s funny how a lot of these different parts of the AI picture hang together.

Bob: Yeah. And actually it leads to a question for Dan. Would it still be a normal technology if you had superintelligence? Is superintelligence part of the normal technology paradigm?

Dan: No. It’s complicated. The way they frame things is they contrast the normal technology worldview with the AI-as-potentially-superintelligent-species worldview. But it’s complicated, because Arvind Narayanan and Sayash Kapoor, the authors behind “AI as Normal Technology”, grant that AI systems could in principle be much more capable than human beings along many dimensions. It’s just that with certain capabilities, like persuading people against their self-interest, and also forecasting, they seem to be quite sceptical. But then there’s this other very strange feature of their worldview, where they think that even if you do think really transformative AGI is possible, we’re not in an epistemic position to forecast what would happen in that world anyway, so we shouldn’t really talk about it. Which makes the whole worldview quite difficult to summarise and engage with. It’s a little slippery, and it bundles lots of different things together.

Negativity bias and the discourse

Dan: Maybe I could talk about one other cognitive bias, because there are a couple of things I really want to get to that have to do with the appendix. You’re really good on tribalism and how it can distort our judgment and perception in counterproductive ways. There’s another bias, negativity bias, and the way it interacts with the incentives of the media and the pundit and public intellectual class. It interacts with tribalism in various ways, but I think it’s one of the most consequential and damaging biases in the modern world. In liberal democracies today you’ve got one of the most peaceful, prosperous societies in human history, and yet large segments of the population basically think they live in a dystopian hellscape, which is why they want to vote for anti-establishment politicians. A lot of the reason for that is the way our evolved tendency to attend to negative and threatening stimuli interacts with the incentives of a very competitive media environment that is constantly broadcasting negativity and threat and outrage. And I worry that this is also happening when it comes to AI.

My view is that at the moment, and we can talk about scenarios and how they’ll play out in future, AI is overwhelmingly a positive technology. I think it’s having very beneficial consequences for the information environment. I think it has really beneficial consequences even when it’s used for things like counselling, to help people with their mental health, and it’s enhancing productivity. And yet when you look at the discourse about AI, it’s so negative, often in quite misinformed ways. Think about the backlash to data centres and all the misinformation about water use. Think about, and this is an area I’ve got interest in and you write about in the book, the doom and gloom about the impact of AI on the information environment, which I don’t think really tracks the reality of how these systems are affecting how people form beliefs and gather information. And I’m a little worried that maybe in the book you might also be skewing too negative, too much in the direction of catastrophe, relative to what the actual reality is. So what do you think about that worry, that we’re too negative, too alarmist, too catastrophising about the technology, relative to what we actually observe?

Bob: So you’re worried that I worry too much?

Dan: Potentially.

Bob: Fair worry. I agree the technology has tons of positive things. I use it, it’s enthralling. That’s one reason I think the current air of opposition to it, at least in America, is a very fluid thing. I don’t know how it’s going to play out. I think a lot of the people who are opposed to it haven’t discovered positive uses they’d discover if they used it more. And all the things it could do, cure cancer and so on, would be great. I actually used it, I had cancer a year ago and I’m fine now, and it came in handy in particular ways, more as a research tool. I don’t doubt it can do a lot of the miraculous stuff.

Now, as for a negativity bias, first of all, as a cognitive bias, I’ve always thought it couldn’t be a quite systematic bias, because I think natural selection designed us to be unduly optimistic in some circumstances. For example, heterosexual males see a woman in the distance and overestimate her attractiveness relative to what they’ll eventually judge when they see her close up. That’s a kind of positivity bias. And there are probably circumstances in which they’re inclined, on average, to overestimate how attractive they’re being found by other people. Consequentially, I think there are certain group dynamics of optimism that are natural and unfounded, and you see that on both sides in the run-ups to war: yeah, we can win this thing. That may have some basis, I’m less sure in evolution, but it’s a thing, you see it all the time. But yes, are people also inclined, quite a bit, to worry about things? Yes, you’d expect natural selection to create animals attentive to threats. Watch a squirrel while it’s eating its nuts, always looking around for stuff to worry about.

As for whether we need more of it or less of it, I don’t know. I can certainly cite examples where I was worried and history bore me out. I was opposed to the Iraq War, and most people in America say, yeah, that was a mistake. I was opposed to the recent attack on Iran; that didn’t work out. America has had so many foreign policy misadventures in my life that were preceded by what in retrospect was undue positivity of a kind. And you can come up with specific examples, like thalidomide, you’re too young, but thalidomide was a drug they gave to pregnant women without enough contemplation. I knew a guy in college who was a thalidomide baby: he was missing the part of his arm between his elbows and his shoulders, his forearms came directly out of his shoulders. So you can come up with examples on either side. I’d just say, if you’re going to have as much positivity coming out of Silicon Valley about technology, and it’s going to be as well funded as it is, you definitely need people like me, just so you can have a debate. You’d agree debate about this stuff is good. You need both kinds of people: optimists to argue with me, and pessimists to argue with them. I don’t know whether you can point to cases where the balance works too much in one direction or the other. That probably wasn’t a satisfactory answer.

Dan: No, it was. I completely agree that sometimes worrying is completely legitimate, and when it comes to AI specifically there are things to worry about. I just think in general there’s a systematic mismatch between how negative people’s picture of the world is and the actual empirical reality. There’s lots of data on this, it’s the kind of thing people like Steven Pinker bang on about. People think crime is increasing; actually violent crime has dramatically decreased. People think back in the good old days we used to be so wealthy; actually we were a lot poorer in the past than we are now, and so on.

Bob: And people think China wants to take over the world. So I’m kind of on the same page, now that I think about it. I’m just saying there are cases where I’d argue the bias is too negative, like China, and cases where other people would argue the other way. I don’t think the world faces some kind of uniform negativity bias. I think people of certain ideological persuasions, very pro-free-market, see a certain kind of negativity, and people of my foreign-policy persuasion see another kind, like threat inflation. And you just need to argue about stuff. As evidence that I don’t think it’s a uniform bias against technology: the social media revolution was not accompanied by this degree of negativity. Not nearly. People were pretty upbeat about it. And the way that played out, and Dan, I know you’re sceptical of the claim that social media had a polarising effect, but for present purposes the relevant point is just that a lot of people think it did, or think it messed up their kids, and they feel like they were burned by Silicon Valley optimism. That’s one thing going on now with AI that informs the reaction against it. But my main point is, I’m older than you, and I’ve been through periods of tremendous technological optimism. It’s not a uniform human thing. So maybe we should pay attention to it when it happens.

Dan: Yeah. The social media thing opens up a whole other can of worms, and we should be time-conscious, so we can’t get into it now.

Does evolution have a purpose?

Dan: I really wanted to end by touching on the appendix of the book, where you have these spicy takes that are likely to interest us as philosophers. I should say the majority of the book...

Bob: I like that word, because otherwise I doubt anyone will read it. Spicy. That’s as sexy as you can make this subject sound. Thank you.

Dan: The bulk of the book is a down-to-earth, concrete analysis of the technology and the political economy. But it’s suffused with something more, and it really gets to it at the end. You’ve got an argument about whether the evolutionary process has a purpose, is directed in some sense, and an argument about consciousness, which I found really interesting. The first argument, about evolution itself maybe having a kind of overarching telos or purpose, do you want to walk us through the basic argument?

Bob: Yeah, first, to emphasise, that doesn’t mean you’re departing from a strict materialist view of the way either biological or subsequent cultural and technological evolution unfolds. Machines can have purposes; material processes can have purposes. And a purpose, a telos, can be imparted in a lot of ways. It can have an intelligent designer in the conventional sense, the way we design machines that have a purpose. But it can also be imparted by what you could maybe call intelligence. There’s an argument that natural selection, for example, well, let’s not go there. Let me just say you could argue that natural selection imparts purpose, or a goal, to organisms, namely to get genes into the next generation. That is the criterion of “design,” and I put it in quotes because natural selection isn’t an intelligent process, isn’t an intelligence in any conventional sense. It’s an information processing system, but it is not, we assume, some kind of conscious designing intelligence, and in any event nothing like the ones we know. People like the late Dan Dennett didn’t think evolution has a purpose. And by the way, I’m strictly speaking agnostic. I don’t know. I do point to things that I think add weight to the case that the unfolding of evolution has some purpose.

What I just said about the sense in which natural selection imparts purposes to organisms means that even if evolution has a purpose, that could have been imparted by a process. I trot out this variant of Lee Smolin’s cosmological natural selection, which involves a replication of universes, according to which, if intelligence in a universe facilitates the reproduction of the universe, then you can imagine Lee’s dynamic leading to it. I’ve discussed this with him on the podcast, and I was happy to find that a couple of people have actually posited that a version of his selective replication of universes scenario, cosmological natural selection, would favour universes that give birth to systems that lead to intelligence. So in this scenario, if you want to play it out, Lee says, well, maybe black holes are portals of replication, in which case that would explain why there are so many black holes, because this process would favour universes with black holes, since those universes are prolific. Well, it could be that superintelligence helps us make more black holes. This is a wild example, of course. So my point is, if evolution has a purpose, it could be an old-fashioned deistic god, could be aliens from another planet, could be something that’s not intelligence in either of those senses.

That said, I try to draw a kind of ironic support from Dawkins’s book The Blind Watchmaker, to a very limited extent. Of course, he didn’t think evolution has a purpose, but his mode of analysis there does at least acknowledge that you can inspect systems for signs of purpose.

Henry: Just to get really clear: there’s absolutely a notion of purpose you can see in biological evolution that Dennett’s really happy with. He uses the kinder vocabulary to describe how evolution kind of aims towards certain goals, and using techniques like dynamical systems theory, talking about state spaces and equilibria, you can say evolution is going to tend towards certain kinds of highly adaptive designs that fit their environments, given certain environmental constraints: multicellularity, flight, vision, and so forth. But it sounds like there’s something a little more heavyweight in your notion of purpose.

Bob: Yeah, well, these are different levels of organisation. Dan wouldn’t say evolution has a purpose. He’d say evolution instils purpose in organisms. The question is, if you move to a higher level of organisation, look, if you step back and look at the three and a half billion years of life, you do see some directionality. You move to higher levels of organisation, richer complexity: multicelled life, societies of multicelled life. One society of multicelled life, ours, launches a second kind of evolution that carries organisation to higher levels: hunter-gatherer village, ancient state, and so on. Now we’re on the verge of globalisation. We’ve built something that looks kind of like a global brain, just with the internet. And, as I note, people were talking about a global brain a hundred years ago; it was discernibly developing a hundred years ago. So it has some things in common with the maturation of an organism, if you step back far enough. That’s what an organism does: it starts out as one cell, and then you get functional differentiation and all these things that I’d argue you see in the course of evolution. I had a whole conversation with Dan Dennett about this, and it’s on YouTube. But I want to be clear: Dan agreed, and he died three or four years ago, that evolution, you might want to put design and purpose in quotes, fine, “designs” organisms, “imparts” a purpose, just getting genes into the next generation. But he wouldn’t say evolution’s purpose is to get genes into the next generation; it’s the purpose instilled by evolution in the animals. And I’m stepping back and saying, well, what if the whole process has a purpose? What if the first life was a seed for the evolution of an organism? I’d say you step back and look at the process, and there’s more evidence than there would have been if nothing particular had happened after the first self-replicating strand of information occurred on this planet and you never got complex cells. Even if multicellularity had evolved only once, but it evolved many times, and that suggests it was likely to happen. So there is evidence you can adduce. But I just want to be clear: the question I’m addressing in the appendix, and I’m not claiming to be confident of an answer, is whether purpose resides at that higher level.

Dan: Sorry to cut you off, I was just going to say, I think it’s a very interesting argument. Where the analogy breaks down a little for me is: if I come across a watch, it shows clear evidence of design, so I want to understand the design process by which it came about. If I come across an organism, again it shows clear evidence of something that looks like design, so we appeal to something like evolution by natural selection that could give rise to that appearance. But if you’re looking at the whole evolution of life on the planet, I agree you’re encountering a lot of complexity, but it’s not clear to me why the entire system itself would scream design of a sort that couldn’t be explained just by the nuts and bolts of evolution by natural selection.

Bob: Okay, let me ask you: what if, instead of looking at a mature organism, we observe an organism throughout its ontogeny, its entire maturation from single cell to organism? You view the way the cells replicate, they differentiate, and you understand the mechanism, and you see that this was all likely. It’s not like, if you took this same cell and put it in the same circumstance, this whole unfolding towards a mature organism would be unlikely, a fluke. And you wind up with this big brain, say it’s a primate, that governs this whole body. You carry the functionality to higher and higher levels, and then you get this coordination of the whole thing via a brain, and you understand that the process was driven by things that made it very likely to happen that way. That’s why, if you’re going to argue about evolution, it does matter that multicellularity evolved many times. Now, people argue that there have been certain thresholds that were unlikely to be crossed. That’s an important part of the argument. There was a book by John Maynard Smith and another author called The Major Transitions in Evolution, and that was their view: there were things that were unlikely to be crossed. Fine. But this argument about how likely we were to get to intelligence, some great biologists agreed with me on it, including William Hamilton, author of kin selection theory. He agreed, and further agreed that it did kind of suggest that maybe aliens planted the life. To him it seemed so likely that one species or another would get to intelligence that maybe it did have a purpose. So he bought not just the premise that the unfolding towards some kind of intelligence had been likely, but that it was some kind of evidence of purpose. That too is on video; it’s in a New York Times piece of mine, the video snippet is called “Does Evolution Have a Purpose,” I think, so you can watch him say it. And my question to you, Dan, is: is that a legitimate part of the consideration of whether the organism has a purpose, the seeming directionality of the unfolding towards this brain that coordinates the whole thing?

Dan: Where I stand, I don’t find it a crazy idea, and I found the argument interesting. But to me there seems to be a big difference. If we think about an organism and that process of development and maturation, it’s culminating in a kind of unified, bounded, purposive system that can survive and reproduce effectively in its environment. Whereas when I think about the evolution of all life on earth, it doesn’t seem to be culminating in a really unified, bounded, coordinated system in the way you find with an individual organism, at least as of now. I mean, it’s part of your argument that you think that’s the direction, that we’re looking like we’re going to culminate in something that looks a lot more like an individual organism. Is that the argument?

Bob: Yeah. I’d say already you could say the global economy is like a global brain; it’s an information processing system that allocates goods and services. But the governance part, the literal coordination, what do brains do? They coordinate bodies. I’m saying that part of it, there’s a part in the final chapter where I say it’s almost like what our species is being told is: look, we can do this global brain thing the easy way, or the hard way. The easy way, compatible with your ongoing welfare, is for you to design a system of international governance that minimises the concentration of power, no more than is necessary, that is democratic and decentralised, and in which you continue to flourish. Or things are going to get ugly. At least in the doomer scenarios, and in the book I look at the escape scenarios, where AI escapes our control or takes over, and find that they’re not crazy, which I thought they were ten years ago. Or maybe something like that happens, or maybe just a human authoritarian takeover by somebody empowered with AI. But I do think that if we do not understand how imperative it is to start coordinating internationally, and to do it in a way I’d like, it may get done in a much uglier way, involving more death and suffering and ultimately less human freedom in one sense or another. So my answer to your challenge is: the story’s not over. I think it’s actually pretty likely that you’ll wind up with some global coordinating mechanism. I want it to be the good kind. But the jury’s out.

I will say, of course, that when you talk about purpose in evolution, you’re working at a handicap, because we only have one data point. When we look at animals, there are tons of members of each species, and tons of species, so it’s much easier to conclude that something gave them the purpose: they all do this one thing, get their genes into the next generation, there must be some reason for that, and that’s what Dawkins would agree with, that part of the analysis. But we only have one instance of the whole process, and that’s why I say, if you’re interested in this question, and a lot of people aren’t, and that’s fine, then you have to inspect the ontogeny, the dynamics of the unfolding and its directionality. That’s the only data you have to work with.

Consciousness and epiphenomenalism

Bob: And then, as you know, I have this very speculative argument about consciousness in the appendix. If we’re out of time I’ll just say, for philosophers, people interested in the mind-body problem: I contend that what we call epiphenomenal consciousness, which could be the kind of consciousness there is, could have a function, if that function were imparted at the higher level of organisation by the designer of evolution, whether that’s a system of intelligence or whatever. But an epiphenomenal consciousness could not be something invented by natural selection on this planet, unless you think it’s of very recent origin. If you think prehuman animals have consciousness and it’s epiphenomenal, it could not possibly have been designed by evolution for a purpose, because it doesn’t start affecting reality until you get animals with language that talk about it, at which point in some sense it ceases to be epiphenomenal. This conversation we’re having is a function of the existence of consciousness, even if consciousness was epiphenomenal for the first however many billion years.

Henry: Just on a point of clarification. When people use the term epiphenomenal, at least in regard to consciousness, there are two main ways they can use it. One is that it’s an evolutionary epiphenomenon, a spandrel, in the biological terminology; it doesn’t have an evolutionary function itself. The other is the more metaphysically laden notion, where it is metaphysically or ontologically epiphenomenal: it’s not within the inventory of the physical, it’s a non-physical phenomenon that supervenes on the physical world. Am I right in thinking you mean the first one, or something in between the two? Some secret third thing?

Bob: Well, I wouldn’t call it, I don’t think, to be an epiphenomenalist, as I understand it, you have to think consciousness itself is physical. You think it is affected by the physical body, by physical information processing in the body, but does not affect that. Again, until we start talking about it, and not just talking about it academically, but saying, I feel bad, you hurt me, that hurt me. Those things change, and that enters into my argument about the sense in which it could be functional at the social level, an integrating force in a sense, but that function would have to have been imparted at a higher level of organisation. There’s another reason I think that, which is that I think of consciousness as an inherently deeply metaphysical thing. It would have to be a property of the universe. The idea that natural selection invented it in the sense that it invented toes is, to me, just crazy. Anyone who says that does not share my understanding of what consciousness is, which of course many people don’t, but I just think consciousness has these completely unique properties. It’s private; no other person can observe it. There’s nothing else in the universe we know of that’s like that. So that’s weird and profound and metaphysical, and I just don’t imagine natural selection producing a mutation that does that. The mutation would have to be at the level of the universe, I think. Like, this is a property of information processing in this universe, kind of.

Henry: So it sounds like the picture you’re broadly sympathetic to is a kind of David Chalmers-style picture, where consciousness is non-identical to any physical property but supervenes on physical properties, thanks to something like brute psychophysical laws that just obtain within this universe: when you get the right kind of collocation of physical structures, you get this further thing that is not itself physical, which is consciousness. Is that a rough reconstruction of the view?

Bob: I think that’s fair. On epiphenomenalism, I actually had a conversation with him on my podcast, and I think we have a slightly different view on how we have to define epiphenomenalism, posed by the fact that it seems to start affecting the world as soon as animals with language start saying things about it. There was a metaphor I didn’t have at my disposal when I was talking to him that I’ll use now. When I talk about a consciousness that was epiphenomenal until people start talking about it, he has trouble with that happening, and I think has to define it away via his definition of epiphenomenalism, or something. No, that’s probably not fair, so dismiss whatever I just said. But a good metaphor for epiphenomenal consciousness, to begin with, is a shadow. My hand moves along, it has a shadow. The movement of my hand affects the shadow; the shadow doesn’t affect the hand. But once I have eyesight and look down and see my shadow, the shadow can affect my behaviour. I think the advent of language, with respect to an epiphenomenal consciousness, is like the advent of eyesight with respect to a shadow: it imparts causal import to a previously epiphenomenal thing. I should say, I’m not a dyed-in-the-wool epiphenomenalist on consciousness. I’m a mysterian; I don’t know what to make of it. Epiphenomenalism is the most intuitively appealing thing, but since that raises the question of what it’s for, my appendix proposes an answer. There’s something it could be for, but only if the functionality was imparted by something that created evolution, in the scenario where evolution has a purpose.

Dan: I’m conscious of the time, but maybe I’ll throw in one quick, potentially clarificatory question, potentially objection, just to get a better sense of your view, and then we can wrap up. I suspect we’d really have to do a whole conversation on this to get into the weeds, but we’ll get you back on. My thought was something like this. You’re an epiphenomenalist...

Bob: I’m available. Not today, but I’m available.

Dan: ...it seems in part because you think you can explain behaviour in general, but you find it at least somewhat plausible...

Bob: First of all, I’m not an epiphenomenalist. It’s a plausible, it’s the most intuitively appealing view of consciousness to me, but I have no idea what consciousness is.

Dan: Fair enough. Okay, but you find it plausible because you think, in general, setting aside the fact that we talk about it, we can explain our behaviour, and neuroscientists often do explain our behaviour, without appealing to consciousness, which is this weird thing. Firstly, I don’t quite agree with that, because I find things like the global workspace theory of consciousness quite plausible. But even setting that aside, if you think that about our behaviour in general, presumably it would also apply to our verbal behaviour. So why wouldn’t you also find epiphenomenalism plausible in the case of us talking about consciousness?

Bob: You mean it facilitates the use of language, or, wait, maybe I missed something.

Dan: Well, the thought would be, as I understand your argument, it’s...

Bob: Why couldn’t it still be epiphenomenal?

Dan: Presumably the same argument would go through. I’m making these sounds...

Bob: As I understand it, it’s part of the definition of epiphenomenal that the thing you’re calling epiphenomenal does not affect the material world. And right now the particular physical sound waves we are exchanging would not be in the form they’re in if consciousness did not exist. If subjective experience did not exist, we would not be having this conversation. Most conversations people have, you can imagine happening even if there were no sentience. You can imagine complex, smart organisms that communicate in as sophisticated a way as we communicate without it being like anything to be them. But you cannot imagine this conversation, us talking about what it’s like to be us, if there were no such thing. And you cannot imagine somebody saying, ouch, you just stepped on my toe and that hurt. You can imagine them saying you stepped on my toe and that threatens to damage my toe, but you can’t imagine explicit references to internal states if they don’t exist. That’s what I’m saying. This problem is acknowledged, and I forget David Chalmers’s workaround, but it’s in recognition of this challenge to epiphenomenalism that he does some complicated thing that, as I recall, kind of defines the problem away. But the straightforward way to describe it is: if consciousness is epiphenomenal, it ceases to be, in the strict sense of not having any effect on the physical world, once you have animals that communicate about sentience, about feelings and thoughts. That’s the argument. Does that make sense or not?

Dan: It makes sense. I think we’re going to have to have you back on for a fuller conversation about this.

Bob: Well, I just said everything I know about it. But yes, I’m game for coming on again in principle, for sure.

Dan: Fantastic. We really appreciate you coming on. I highly recommend the book. I think it’s one of the best books on this subject, one that people will get a lot out of even if they’re already immersed in the literature. Henry and I will be back in a couple of weeks. So thank you, everyone.

Henry: Thanks, everyone, and thanks, Bob.

Bob: Thank you. This was a great conversation. I really appreciate it.

Do citizens live in “different worlds”?

In recent years, there has been considerable scholarly and political attention to the worry that citizens of many democracies increasingly inhabit different “worlds”. This fear is acute in the US, where sharp polarisation between Democrats and Republicans is intense and seemingly ever-increasing, but it is also salient in many other Western countries where party-based identifications are swamped by other divisions, such as left versus right, cosmopolitan versus nationalist, establishment versus populist, and so on.

Concretely, the worry is that on topics as diverse as election integrity, immigration, crime, vaccines, and the Epstein files, many citizens do not just hold different values and interests but operate with fundamentally different factual understandings of what is going on.

This creates an obvious problem. Democracies can function if citizens have different experiences, interests, values, and ideologies, but if they can’t even agree on what is happening in the world, we are in trouble.

As Obama said, “We want diversity of opinion; we don’t want diversity of facts.”

This worry is typically coupled with a familiar partisan interpretation. For liberals and progressives like Obama, the fundamental problem is not so much that citizens inhabit distinct realities as that some citizens confront reality whilst others—the right, or at least the populist right—inhabit a fantasyland populated with fake news, misinformation, disinformation, demagogic lies, and conspiracy theories.

For the right, this interpretation is reversed. As Rush Limbaugh declared in 2009, “Everything run, dominated and controlled by the left here and around the world is a lie. The other universe is where we are, and that’s where reality reigns supreme and we deal with it.”

The suboptimal nature of this situation has fuelled much scholarship, discourse, and policy aimed at diagnosing what has gone wrong and prescribing various cures.

However, a prior question is whether this assessment of substantial “factual polarisation” is accurate to begin with. How widespread is this phenomenon, really? How worried should we be?

Subscribe now

Three sources of scepticism

One source of scepticism is a now-familiar body of research alleging that post-2016 worries about fake news and misinformation are overblown. According to this work, clear-cut falsehoods and fabrications are less prevalent in the information environment than many assumed.

More relevantly, there is also a persuasive body of research suggesting that estimates of factual misbelief and polarisation have been systematically overstated due to flawed survey designs. It turns out that when you pay respondents to give accurate answers (to guard against insincere partisan “cheerleading”), include “don’t know” options, and screen for inattention and trolling, the amount of outright factual errors and disagreements in public opinion tends to drop quite substantially.

Finally, whereas five or ten years ago, it was all the rage among social scientists and public intellectuals to argue that people are so irrationally tribal they are often immune to facts and sometimes even “backfire” when confronted with corrections, such ideas have shared the fate of countless other sexy social-psychological findings: they generally haven’t replicated. The more robust, boring finding from recent research is that when people are presented with factual evidence and rational arguments, they tend to update their beliefs, even when that means moving away from their tribe’s orthodoxies.

One reasonable lesson you might draw from all this is that the “different worlds” panic is unfounded. Perhaps most citizens are more closely tethered and responsive to a shared reality than was previously feared. They might still dislike members of other political tribes for dumb—well, tribal—reasons, as research on affective polarisation or political sectarianism suggests, but they don’t disagree about the fundamental nature of reality as much as many feared.

A different view

I partly agree with this interpretation. The “different worlds” thesis has been overstated, even in highly divided countries like the US, especially among most normie voters who don’t pay much attention to politics because they have better/more enjoyable things to do. I also think that, as with the misinformation panic, too much of the discussion here has suffered from recency bias, exaggerating how novel the core phenomenon really is.

Nevertheless, I think the core phenomenon is real and important for thinking about some of the central political challenges we confront today. However, much scholarship and discourse has misrepresented it. To understand in what sense citizens inhabit different worlds, we need to move away from differences over narrow, verifiable facts. Instead, the action resides in the competing systems of interpretation through which people select, categorise, frame, connect, explain, and narrate those facts.

Although I am tedious in my repeated emphasis of this point, I think the best account of this phenomenon comes from one of my intellectual heroes, Walter Lippmann, in his analysis of what he called the “pseudo-environment”: the highly selective, low-resolution compressions of reality that most citizens confuse for reality itself. This analysis of the pseudo-environment spotlights a form of political division—of polarisation—that has been overlooked in much (but not all) scholarship: interpretive polarisation. (Neta Kligler-Vilenchik and colleagues call a similar phenomenon “interpretative polarization”.)

Interpretive polarisation is not primarily disagreement over narrow, verifiable facts, nor disagreement over the values or high-level ideologies citizens bring to politics. It involves competing systems of interpretation that determine which facts citizens attend to, how they understand them, and how they connect their values and ideologies to political action.

Part of my aim here is to make the case for this thesis. But I also want to argue that this analysis helps to illuminate some of the most pressing challenges liberal democracies confront today.

For that, it will be helpful to start with one of the biggest stories in recent British politics: the murder of eighteen-year-old student Henry Nowak, and the processes through which the facts of his death were refracted through the rival systems of interpretation that organise political attention and conflict in the UK and broader Western world.

“I can’t breathe”

Many of the facts concerning what happened to Henry Nowak are not in dispute, in part because the final moments of his life were captured on horrific body-cam footage.

Last December, Nowak, a white eighteen-year-old university student, was stabbed multiple times by a Sikh individual named Vickrum Digwa. As Nowak lay dying, Digwa’s brother phoned the police and claimed that Nowak had racially assaulted his brother. When the police arrived, Digwa and members of his family reinforced this story as Nowak lay on the floor, barely able to speak or move.

He told the police officers he had been stabbed. One responded, “I Don’t think you have, mate.”

He was then placed in handcuffs and was initially ignored and dismissed as he said, multiple times, “I can’t breathe.” The officers told him he was being arrested for assault and read him his rights as he lay dying. “Please, brother, I can’t breathe” were some of his final words.

Shortly afterwards, the police officers discovered that Nowak had in fact been stabbed, and Digwa was charged with murder. A couple of weeks ago, Digwa was found guilty of this charge, after which the horrendous body-cam footage from the police officers who arrived on the scene was released to the public, fuelling a vast amount of media attention and political discussion of the incident.

Observing this discussion, it would be reasonable to conclude that many citizens inhabit different worlds. And yet, this manifested much less through disagreement on the verifiable facts of the case than in how they were interpreted.

Similar facts, different interpretations

Among the populist and far-right, the events were imbued with maximal political salience—not as an isolated or unrepresentative occurrence, but as a national emergency, a cause for a political reckoning.

It was categorised as both an instance and proof of how the UK operates with a “two-tier” culture and justice system, in which establishment institutions apply different standards to white (increasingly capitalised “White”) citizens than to racial minorities.

The police’s response was explained not just in terms of incompetence but in terms of the insidious influence of anti-racist/DEI/woke ideology throughout our institutions, which led the police officers arriving on the scene to treat an accusation of racism “more seriously than an act of murder.”

In this way, the events were also filtered through counterfactual assumptions about how such an event could never have occurred had a white man stabbed a Sikh teenager and then falsely claimed to have been racially assaulted.

Most fundamentally, the events were located in a clear narrative of villains, victims, and heroes. The direct victim, Nowak, was a synecdoche for the broader population of ordinary white Britons victimised by liberal elites and racial minorities. To fight back, these victims must rely on brave, truth-telling individuals—politicians, pundits, activists, ordinary men and women—unafraid of being called racist.

This interpretation clashed with the mainstream liberal and progressive reading. Here, the events of Nowak’s murder and the police response were largely treated as an isolated tragedy, unrepresentative of, and so uninformative about, anything broader. Hence, they should not be “politicised” (i.e., treated as carrying information about matters of broad political interest). In contrast, the right’s reaction to such events was assimilated to a broader, more familiar template: far-right demagogues and deplorables exploiting unrepresentative tragedies to whip up hatred and division.

In this narrative, the victims were the racial minorities targeted by such racist hatred, as well as the establishment institutions that function to protect them.

An interpretive inversion

Importantly, one reason the events ignited such a political and media storm is a specific phrase Nowak uttered: “I can’t breathe.”

Back in 2020, these words were uttered by George Floyd as he was killed by Derek Chauvin, events captured in equally horrendous video footage.

In the aftermath of those events, the systems of interpretation that I have described were inverted. Among liberals and progressives, not just in the US but across the broader Western world, Floyd’s death was imbued with maximal political salience. It was not an isolated or unrepresentative murder and tragedy but the cause for a national reckoning—in fact, an international reckoning.

The core narrative frame here was racism—not just Chauvin’s racism, but also the deeper, even more insidious forms of racism (structural, systemic, implicit) that had long poisoned Western cultures and institutions, which Floyd’s death was an awful symptom of. Of course, this causal interpretation implied a counterfactual assumption: that had Floyd been white, this would never have happened.

Most generally, the events were narrated through a story of victims and villains in which Floyd’s direct victimhood both demonstrated and reflected the broader oppression of all black (increasingly capitalised as “Black”) people and other non-white groups around the Western world, who now depended on the heroic actions of progressive, anti-racist activists awake to the reality of such oppression.

Conservatives and right-wing populists viewed the events through a very different interpretive framework.

Most obviously, they resisted any treatment of what they deemed a rare and unrepresentative act of police misconduct as the basis for a broader condemnation of American policing, let alone Western societies as a whole.

They emphasised the specific facts of the case, including Floyd’s criminal record and behaviour. They expressed agnosticism or outright scepticism about the relevance of racism to the events, and often made efforts to publicise similar cases in which white victims were killed by police without receiving comparable media or political attention.

Finally, this media and political attention was itself explained in terms of a broader, familiar pattern: the left’s cynical exploitation of isolated, unrepresentative incidents to advance its hysterical ideological agenda.

For the populist right today, the profound differences in how Floyd’s and Nowak’s deaths were treated provide yet further evidence for their own interpretation. When “career criminal George Floyd” was murdered, the establishment responded by politicising it to an extreme degree, fuelling the “Great Awokening” that swept Western institutions in the years afterwards. In contrast, that same political establishment’s response to the Nowak event is to not politicise it—“proof, if ever there was any, that we’re living in a two-tier culture in this country, where the rights and privileges of white people matter less than those of ethnic minorities.”

For liberals and progressives, this very narrative is itself further confirmation that the right inhabits a deranged fantasyland. Floyd’s death reflected the extreme, well-documented historical oppression of Black people, which contrasts with the delusional victim complex of deplorable, racist white citizens today.

This is what interpretive polarisation looks like.

A Lippmannian analysis

To understand what is going on in these cases, focusing on narrow disagreements over verifiable facts won’t get us very far. That’s not to say that there are no such disagreements, or that nobody is deluded or lying about basic facts. The point is rather that we need a richer account of how such facts become selected, ignored, and narrated in the minds of those who identify with different political tribes.

For the basics of such an account, we can turn to Walter Lippmann’s 1922 masterpiece, Public Opinion.

Public Opinion’s central thesis is simple and, on reflection, obvious.

In contrast to the relatively small-scale social worlds our species engaged with for most of its history, the modern world is far too big, complex, changing, and inaccessible for any individual to engage with directly or completely. Most events, trends, regularities, and public affairs are remote. They happen in places we have never been, involve people we have never met, are affected by complex systems and institutions we cannot fully understand, and turn on countless facts we cannot personally verify.

This has two interacting implications.

First, we rely on others (journalists, pundits, politicians, activists, intellectuals, Substackers, etc.) to mediate reality for us, all of whom are in exactly the same situation.

Second, this mediation cannot involve transmission of a complete set of facts. There are too many facts and too many ways to interpret, connect, and explain them. So, we must reduce reality to extremely selective, low-resolution mental models.

This thesis underlies Lippmann’s distinction between what he calls the “real environment”—the vast, complex, independent reality in which our actions have consequences—and the “pseudo-environment”: our simplified, selective pictures of reality we instinctively confuse for reality itself.

For Lippmann, we cannot understand democratic politics and its defining pathologies without attending to the distinctive characteristics and failure modes of these pseudo-environments. We also cannot understand political divisions and the ways in which different citizens “live… in different worlds.”

A central insight of Public Opinion is that pseudo-environments are not best understood as either a database of factual beliefs or as a judgement on a pre-existing set of facts. Instead, they are constituted by what Lippmann called “stereotypes”, by which he meant not just prejudiced generalisations about social groups but something much broader: all simplifying systems of categories, frames, prototypes, narratives, and explanatory models through which citizens compress reality in ways that make it intelligible.

Although these stereotypes tend to distort citizens’ understanding of the political universe (more on this below), they are unavoidable. Without extreme selection, simplification, and categorisation, political reality would be an unintelligible blur.

Most interestingly for our purposes here, this basic model of political psychology leads him to criticise what he calls the “orthodox theory of public opinion”, which in many ways still shapes how many people think of politics and political disagreement today:

“The orthodox theory holds that a public opinion constitutes a moral judgment on a group of facts. The theory I am suggesting is that … a public opinion is primarily a moralized and codified version of the facts … The pattern of stereotypes at the center of our codes largely determines what group of facts we shall see, and in what light we shall see them.”

Attention and Interpretation

There are two aspects to this model.

The first concerns selection and salience. Neither the mind nor the media can simply hold up a mirror to reality. There are too many facts, too many things going on, so we must be selective in what we attend to and deem newsworthy. We must decide which events matter, which events are representative, which reports are trustworthy, and which things can be ignored as noise.

This means that distinct pseudo-environments could, in principle, emerge not initially through disagreements over narrow, verifiable facts but through different decisions about which facts to attend to and which to ignore.

The second function of stereotypes concerns how facts are categorised, framed, narrated, and explained. The same events can be interpreted as a protest, a mostly peaceful protest, a riot, a coup, an insurrection, a false flag, and so on. The same murder can be interpreted as a local tragedy, a hate crime, a symptom of racism, an instance of two-tier justice, or an anecdote exploited by cynical political entrepreneurs.

We do not first see, then define…

These ideas lie behind perhaps the most famous line from Public Opinion: “For the most part we do not first see, and then define, we define first and then see.”

The point is not that pseudo-environments involve simple hallucinations or that citizens can never perceive anything accurately. It is that how people experience and understand the political universe is shaped by socially supplied systems of categorisation and interpretation.

Public opinion is, hence, not a matter of evaluating a pre-existing, self-interpreting set of facts. It involves applying a pre-existing system of interpretation to the task of selecting, omitting, categorising, and framing facts.

For Lippmann, this notion of “seeing” includes not just what we encounter second-hand through media reports but also our first-hand experiences.

This is why Lippmann is so sceptical of what is now called “lived experience” as a basis for political insight, an aspect of his model of politics that my students tend to find most confusing and maddening when I teach Public Opinion. One’s lived experiences do not come pre-interpreted. Which experiences we attend to, how we categorise them, and how we connect them to broader frames, narratives, and explanatory models is determined by our system of stereotypes, not the experiences themselves.

“I don’t think you have, mate”

One couldn’t find a clearer or more disturbing illustration of these ideas than the case of Henry Nowak’s death.

First, there were the police officers who arrived at the scene and initially dismissed Nowak’s claims that he had been stabbed and, when arrested, could not breathe.

They did not first take in the facts and then form a judgement. They seem to have arrived with a pre-existing interpretation, which guided what they saw: a drunk, racist white guy lying on the floor, and a victimised Sikh family. It has even been said that Nowak’s uttering of “I can’t breathe” might have been interpreted as further confirmation of this interpretation, precisely because in the aftermath of Floyd’s death this has become a slogan often mockingly used by the far-right when arrested by police.

Then, of course, there is the politics of the event, the competing systems of interpretation highlighted above, in which the body-cam footage was encountered and understood through competing systems of salience, categorisation, and explanation.

To understand this, we need to identify a distinctive form of polarisation, one which is not well-understood in terms of traditional categories, including factual polarisation (disagreement over narrow, verifiable facts), ideological polarisation (disagreement over high-level values, principles, or ideologies), or affective polarisation (mutual dislike among members of opposing groups). Interpretive polarisation is connected to all of these—competing systems of interpretation will typically co-exist with these other forms of polarisation—but it is not reducible to them.

It has many dimensions, including categorisation, causal and counterfactual models, representativeness judgements, moral framing, and narratives and narrative roles, all of which mediate between the facts people attend to, ignore, and explain, as well as the way they apply high-level values (e.g., concerns with fairness, equality, or freedom) to politics.

Importantly, this is not a new proposal, not just because it is in Lippmann’s work, but also because many other researchers have expressed similar insights. Nevertheless, I think the core idea is underrated in how many people think about politics, polarisation, and public opinion.

Some Implications

There are several reasons why this matters.

First, we shouldn’t move too quickly from findings that simple factual mistakes and polarisation are rare to the lesson that citizens confront a shared reality. Radically different systems of interpretation can co-exist with a shared recognition of narrow matters of fact. In fact, just as the most effective propaganda rarely involves outright falsehoods, the most resilient systems of “stereotypes” are precisely those that are difficult to falsify.

Second, we also shouldn’t move too quickly from findings that clear-cut “misinformation” and misperceptions are relatively rare to the conclusion that most citizens have an accurate understanding of reality. Lippmann was concerned about pseudo-environments precisely because they typically simplify and distort reality in biased, cartoonish ways that fuel bad politics.

For example, the increasingly popular conviction that white (“White”) Britons are victims of an oppressive two-tier justice system is not easily subject to traditional “fact-checking”. Of course, one can point to many facts that seem to contradict it (e.g., the dominance of white people among the higher strata of British society), but one can also point to facts that seem to confirm it (e.g., laws and policies that often involve de facto positive discrimination in favour of non-white groups). At bottom, however, it is not a simple, verifiable claim about reality. It is a highly charged interpretation of reality capable of gobbling up confirming information and screening out disconfirming information.

For most of those who live in this pseudo-environment, it doesn’t even exist as a fully articulated set of propositions. It involves inchoate feelings of identity, status, and grievance coupled with a high-level narrative, the precise details of which are then fleshed out, connected to ongoing events, and defended by a professional class of politicians, pundits, and activists who work to construct and update the details of the pseudo-environment in real time.

Nevertheless, the core narrative is a gross simplification and distortion of a far more complex reality. It transforms a scattered set of real failures and asymmetries into a clean narrative of anti-white institutional oppression, omitting any facts and history that don’t fit this story, which is framed, exaggerated, and embellished in terms of stick-figure categories of villains, victims, and heroes.

Of course, such distortions are not unique to this pseudo-environment. The Great Awokening that swept Western institutions in the aftermath of Floyd’s murder also stitched together a complex set of truths, omissions, simplifications, exaggerations, homogenisations, and Manichean distinctions. (This was especially noticeable in the UK, where a narrative adapted to the US context was often applied here with almost no modifications, despite the radically different history of race, immigration, and multiculturalism in the country.)

More generally, Lippmann’s point was that although not all pseudo-environments are equally wrong, they are all, in a sense, wrong.

Partly, this is because such pseudo-environments are never neutral nor disinterested. A pattern of stereotypes, writes Lippmann, is “the guarantee of our self-respect; it is the projection upon the world of our own value, our own positions and our own rights.”

As strategic political primates, we instinctively gravitate towards interpretations that frame and narrate reality in ways that advance our interests, flatter our allies, demonise our rivals, and mobilise support for our favourite causes. The construction of a pseudo-environment is, therefore, a complex move in a society-wide status game, a distributed attempt to interpret and misinterpret reality in ways that assign the right groups, organisations, and institutions to the categories of villains, victims, heroes, dupes, cowards, hypocrites, and demons.

This constructive process is also—and this is perhaps what most troubled Lippmann—a profoundly unscientific endeavour. By the early twentieth century, it had begun to dawn on most intellectuals that studying society effectively would require applying rigorous scientific and statistical methods for data collection, inference, and hypothesis testing.

In an amusing passage, he writes,

“To pick fairly a good sample of a large class is not easy. The problem belongs to the science of statistics, and it is a most difficult affair for anyone whose mathematics is primitive, and mine remain azoic in spite of the half dozen manuals which I once devoutly imagined that I understood. All they have done for me is to make me a little more conscious of how hard it is to classify and to sample, how readily we spread a little butter over the whole universe.”

And yet, how are pseudo-environments constructed? Even setting aside the distorting role of people’s interests and self-deception, systematic inquiry into a complex, distant reality is replaced with pre-scientific intuitions, the limbic system, personal experiences, and headlines. (Today, we might also add viral clips and hot takes.) In place of representative samples, base rates, and reliable causal inference, we get the modern equivalent of primitive myths, except instead of guiding rain dances, they power our democracies.

Looking around at today, it is hard to avoid the conclusion that Lippmann’s worries about democracy and public opinion were well-founded. In fact, it is noteworthy that as liberal democracies have become more democratic, especially as social media has weakened the influence of establishment gatekeepers, the political influence of pre-scientific, primitive myth-making on both the far left and populist (increasingly outright fascist) right has gotten far worse.

What can be done?

At least when writing in 1922, Lippmann favoured a broadly technocratic solution. If the problem is that public opinion is pre-scientific, the solution must rest with “organised intelligence”: institutions that apply rigorous scientific and statistical methodologies to classify, analyse, and explain the social world.

In the second half of the twentieth century, this broadly technocratic impulse became highly influential across Western societies. As Benkler and colleagues observe,

“Government statistics agencies; science and academic investigations; law and the legal profession; and journalism developed increasingly rationalized and formalized solutions to the problem of how societies made up of diverse populations with diverse and conflicting political views can nonetheless form a shared sense of what is going on in the world.”

However, the two cases reviewed in this post show why this solution is so fragile.

First, the ostensibly neutral knowledge-producing institutions—the “organised intelligence”—championed by Lippmann can themselves become captured by popular pseudo-environments. This was evident in the Great Awokening, during which major knowledge-producing organisations and institutions across Western societies increasingly subordinated traditional epistemic goals—truth, objectivity, knowledge, understanding—to the project of social justice as conceived within the progressive pseudo-environment.

Of course, within this pseudo-environment, the very idea of “neutral” truth-seeking institutions was itself framed as a cover for the biases and interests of the dominant (white male cis heteronormative etc.) class.

Whatever one thinks of that debate, it illustrates the challenges of maintaining technocratic institutions in democratic societies with sharp divisions.

It also highlights a second issue: even if such institutions were perfectly objective, they could only shape public opinion if people trust them. If citizens inhabit a pseudo-environment in which such institutions are viewed as captured, corrupt, or biased, they cannot perform the role that Lippmann hoped for.

This basic problem was evident in the reaction to the Nowak case: when the narrative is that mainstream institutions are biased against white people due to the insidious influence of woke/DEI ideology among professional-managerial elites, it becomes difficult to rely on those very same elites to evaluate their own bias.

Again, what can be done?

We can gesture at some things that might help.

Most importantly, we should improve the reliability and impartiality of our epistemic institutions, which are too often politicised and advocacy-focused. Ideals of perfect objectivity and neutrality will never be fully achieved, but they are not simply a cover story for hidden interests either. The left’s growing institutional dominance in recent decades, combined with a prominent strand of left-wing thought that seeks to replace norms of objectivity with social justice activism, has had bad effects on these institutions’ quality and trustworthiness.

Representatives of these institutions also need to be better at adapting to the constraints and incentives of the new media age. The days of relying on establishment gatekeeping and elite control of the public narrative are long gone. We now live in a more fragmented, more competitive attention economy, one which favours more direct, authentic modes of communication. As Renée DiResta and Rachel Kleinfeld argue, “Organizations and scholars grounded in fact-based argumentation—and the philanthropists and advocates who support research-backed policy influence—must grapple with this shift.”

Finally, I am cautiously optimistic that advances in AI might help with some of the challenges described in this essay, but that is a story for another day.

Ultimately, however, it’s easier to diagnose the disease than to prescribe the cure. Freed from the constraining influence of reliable, expert knowledge that only systematic inquiry can provide, democratic citizens will construct rival realities, each a grossly simplified and distorted map of a complex environment that they will confuse for the environment itself.

Further Reading

• My essays, The world outside and the pictures in our heads, Let’s not bring back the gatekeepers, Did social media destroy democracy—or give it to us good and hard?

• Walter Lippmann, Public Opinion

• DiResta, Invisible Rulers

• Neta Kligler-Vilenchik et al., Interpretative Polarization across Platforms: How Political Disagreement Develops Over Time on Facebook, Twitter, and WhatsApp

• Will Storr, The Status Game, on the role of status in constructing narratives and much more

• David Pinsof et al., Alliance Theory of Political Belief Systems

• Jonathan Rauch, The Constitution of Knowledge

• Jeffrey Friedman, Power Without Knowledge

In this episode, Henry and I speak with Benjamin Todd, co-founder of 80,000 Hours and author of the new book 80,000 Hours: How to Have a Fulfilling Career That Does Good. Benjamin has become one of the best writers on AI, producing highly informative essays such as “Will we have AGI by 2030?”, “How AI-driven feedback loops could make things very crazy, very fast”, and “How not to lose your job to AI”.

Among other things, we discuss:

• Why Benjamin thinks artificial general intelligence by around 2030 is a serious possibility.

• Why the most important question is not just whether models get bigger and better in general, but when new feedback loops kick in.

• The difference between impressive benchmark performance and real-world economic usefulness.

• Whether artificial intelligence research is easier to automate than ordinary white-collar work.

• Chip bottlenecks, data centres, talent constraints, and the geopolitics of Taiwan.

• Whether artificial intelligence will cause mass unemployment, and why “become a plumber” is probably not good career advice.

• Why 80,000 Hours has shifted so much of its attention towards artificial intelligence.

• Alignment, control, concentration of power, engineered pandemics, and other risks.

• Whether effective altruism and 80,000 Hours are making a dangerous reputational bet on short artificial intelligence timelines.

• The worry that much short-timelines thinking comes from a small, socially connected, and homogeneous intellectual community.

This was one of my favourite conversations on this podcast so far. It’s pretty dense and high-level in parts. (I would recommend reading some of the essays we link from Benjamin if you struggle to follow parts of the discussion). But if you’re interested in hearing some of the most persuasive, evidence-based arguments for why the world might be about to become extremely crazy, extremely fast, I highly recommend it.

Links and further reading

1. Benjamin Todd’s website and Substack.

2. 80,000 Hours

3. 80,000 Hours: How to Have a Fulfilling Career That Does Good — Benjamin’s new book.

4. “Will we have AGI by 2030?” — Benjamin’s case for taking short timelines seriously.

5. “How AI-driven feedback loops could make things very crazy, very fast” — the most directly relevant essay for the “explosive progress” theme.

6. “How not to lose your job to AI” — Benjamin on careers, automation, and which skills may become more valuable.

7. “Shortening AGI timelines: a review of expert forecasts” — useful background on how forecasts have changed.

8. “Are the last 3 months the start of an AI acceleration?” — Benjamin’s more recent reflections on whether progress is already speeding up.

9. Dwarkesh Patel, “Thoughts on AI progress” — includes the line we discuss in the episode: are models getting more impressive at the rate short-timelines people predict, but more useful at the rate long-timelines people predict?

10. Matt Reardon, “If Anyone Bursts It, Effective Altruism Dies” — a sharp piece on artificial intelligence hype, effective altruism, and reputational risk.

11. AI 2027 — the influential scenario forecast that has shaped a lot of recent discussion about short timelines and intelligence explosions.

12. 80,000 Hours, “How to get into AI safety in 3 months” — a practical guide for people who want to work on reducing risks from advanced artificial intelligence.

Transcript

• Please note that this transcript has been lightly AI-edited and may contain minor errors.

Introduction

Dan: Welcome back. I’m Dan Williams, back with my co-host Henry Shevlin. Today we’re honored to be joined by Benjamin Todd. Benjamin is co-founder of 80,000 Hours, an enormously influential organization dedicated to helping people have the most impactful but also rewarding careers possible over the roughly 80,000 hours that make up a typical working life. He’s also the author of a brand new book conveniently titled 80,000 Hours. If you’re watching this, you can see me putting it in front of the camera. I highly recommend that people buy it and read it. And finally, he has one of the very best Substacks in the world for learning about AI, the key developments and risks, and how to navigate our transition, both as individuals and as societies, to a world after truly, radically transformative AI, which is going to be the primary focus of our conversation today. Benjamin, welcome to the podcast.

Benjamin Todd: Thanks so much for having me.

Dan: The thing we really want to focus on is how you’re thinking about AI, how things are going to develop over the next five or ten years, and how we should navigate that. But it would be helpful to start with a brief primer on what 80,000 Hours is, why you started the organization, and why you’ve recently published this book.

What 80,000 Hours does, and the pivot toward AI

Benjamin Todd: The idea is that your career is the biggest decision you’ll ever make, especially for your impact on the world. It’s your biggest resource for helping others and doing good. But there’s very little advice on which careers are actually worth taking in the first place. Most careers advice is just about how to apply to consulting jobs, or how to become a lawyer, but not really which paths are worth going down in the first place. So there’s this enormous opportunity that maybe millions of people are squandering. 80,000 Hours tries to solve that by providing free careers advice. We have online research, a podcast, a job board with a thousand open opportunities, and one-on-one advising.

Dan: I think of it as a great example of evolutionary mismatch. Up until about five minutes ago in human history, the idea that you’d need to think rationally about what to do with your career wouldn’t have made much sense. Now we’re in conditions where it’s hugely important, but we can’t just rely on evolved adaptations to make those decisions rationally, so you need this cultural technology of 80,000 Hours to guide people through it. Another thing worth flagging before we segue into AI: my sense as an outsider is that recently 80,000 Hours has really pivoted to focus centrally on AI, away from the more generalist advice package it was giving before. First, is that impression correct? And second, what drove that change?

Benjamin Todd: It’s partly correct. In the book, a lot of the advice is totally general-purpose careers advice. It’s about the heuristics and frameworks to use when choosing between options, and very practical things like how to actually do a job application and interviews. So that applies to everyone. But then there’s a chapter where we talk about which problems are most pressing in the world, where we present our own view. And for things like the one-on-one advice, because that’s very expensive to deliver per person, we tend to focus it on the problems we think are most impactful to help people enter. So it depends on which part of the organization. But in general, yes. We’ve had AI risks at the top of our list of most pressing problems since before 2016, so for more than ten years.

One big reason is that the timeline to AI has got shorter and shorter, so it’s become a greater and greater priority as that’s happened. The other big reason is that, although it’s a lot less neglected than before, it’s still extremely neglected compared to most other problems people talk about. The number of people working on AI alignment risks is maybe one to three thousand people in the world, even though that might be one of the biggest existential risks we’re ever going to face. And then there are all these other AI risks that have come up recently, like concentration of power, what we’re going to do about digital sentience that you had a great interview about, gradual disempowerment, and so on. These literally have like five or ten people working on each one. So you might think that as AI gets closer it becomes less neglected, and it kind of does, but not enough to offset the huge increase in urgency.

Henry Shevlin: It might come as a surprise to some listeners that most people in the world are not really concerned about AI, given how unrepresentative our listener base probably is. But I completely agree, Benjamin: if you look at the top five concerns in pretty much every Western democracy, AI almost never makes the cut. Very few people are working on this, even though the three of us and our listeners live in these bubbles where 80% of our news content is about AI. Before we drill down on AI, I wanted to ask one other quick thing.

Personal fit, rare skills, and choosing a career

Henry Shevlin: For a lot of people, intuitively, when it comes to choosing a career there’s a tendency to prioritize rare skills. If I could be the best medieval-style flute player in the world, and I’ve got a rare talent for that, there’s a sense a lot of people feel that it would be a terrible shame not to use it, even though the actual positive impact, financial earnings, and maybe even happiness associated with it are less than something more mundane, like becoming a solicitor or an accountant. I’m curious whether you think that’s a correct diagnosis of one of the problems people face when choosing careers, and whether you think there’s any reason to prioritize rare skills.

Benjamin Todd: We talk about personal fit, and there’s a chapter about this in the book. You can think of it as just how good you’ll be compared to the average in a certain career path. In a way it’s the most common advice, the advice my dad gave me, which is “do what you’re good at.” But despite that, people do still somewhat underrate the importance of it. Partly there are these heavy-tailed differences in outcomes. There’s a famous study of expert performance, I think by Simonton, and he finds that something like 50% of the contributions in a field come from the top 10% of contributors. So there are huge differences in people’s fit between different career paths.

The second point is that if you’re at the top of a really impactful field, obviously that can be really impactful. But even if your field isn’t directly impactful, just being successful at anything tends to give you a lot of options to have an impact indirectly. That could be through donating, or changing the conversation about an issue, or meeting other influential people and spreading ideas through them. These indirect ways of contributing tend to get totally neglected in the normal ethical careers advice, which is basically that you should do a helping career, like being a doctor, a teacher, or a charity worker, and that those are the most impactful jobs. But to a large extent it’s much more about how you use the position you have than the specific job title. That said, there is a trade-off. Some careers do give you more opportunities to help than others. If you’re a super niche medieval historian, there is definitely a trade-off there. So you basically have to weigh the two, and it depends on what other options you have.

The case for AGI by 2030: four drivers of progress

Dan: If people are interested in the general 80,000 Hours package, how to think about this in a rational, evidence-based way, there are loads of resources online, plus the book. Let’s pivot to AI. We can come back later to how thinking about what’s going to happen should influence people’s career decisions. You mentioned that timelines to transformative AI now seem alarmingly compressed. It all seems quite imminent in a way it maybe didn’t ten or fifteen years ago. You’ve got a very interesting essay, called different things online, but on your Substack I think it’s “The Case for AGI by 2030.” Walk us through the argument.

Benjamin Todd: There are a lot of different ways of thinking about timelines. The one I take in this article is essentially trend extrapolation, but taking an extra step and asking what the underlying drivers of these trends are and how long we should expect them to keep working. I break it down into four main drivers. There’s increasing pre-training. There’s doing more RL, reinforcement learning. There’s doing more test-time compute, so running the models for longer. And then there’s building these agent scaffolds and having the models go and complete real-world tasks, where you’re also doing RL on whether they succeeded or not. Then there’s an underlying thing driving those four, and potentially new drivers, which is the scale-up of computing power and research labor going into AI.

The basic argument is that all four of these drivers are set to continue, as are the underlying drivers of compute and labor. So we should expect AI progress to continue pretty rapidly, maybe at a similar pace to recently. There are some reasons to think it could accelerate further, and some reasons to think it will eventually slow down, though I’d argue that’s probably after 2028 before it kicks in. We still don’t know where it will end up or how good it will get in a given time frame. But you can at least build a case that there will be large further advances, and it’s plausible you get to something like pretty close to AGI itself, or close to an AI that can help accelerate AI research, or just an AI that can produce enough revenue to keep the whole machine going so that you eventually get there.

I don’t necessarily think it will be before 2030. I’ve actually changed the title now to “Will We Have AGI by 2030?” I do think that’s a very realistic possibility, though I also think it’s very realistic that it’s at some point in the 2030s. And there’s still some chance it’s even beyond 2040 or beyond 2050, though that seems relatively less likely to me at this point.

Henry Shevlin: Out of interest, how do you operationalize the arrival of AGI? A lot of seemingly clear things, like the Turing test, are kind of gone now, but it’s probably not going to be one moment we can look back on and say “that was the day the Turing test fell.” Do you think AGI is going to be similar, or is there a clear landmark?

Benjamin Todd: No, I think it is just a spectrum. I have a post about this on Substack. This came out of DeepMind and Shane Legg. One way of defining AGI is that there are all the possible tasks you can do, and then there’s how well you can do each task, which is the strength of your capabilities, and then there’s how broad a range of tasks you can do, which is the generality of your capabilities. AGI is just AI that’s becoming more general and more capable. Where you eventually draw the line and say “this is AGI” is arbitrary, though people normally draw it around the level of a human worker: you can do about the same range of tasks about as well as a human worker, for the same cost.

The reason people choose that is because once you automate human labor you can get a type of feedback loop that could lead to transformative change of the economy. Though there are other points at which you might get transformative change well before that. So from a day-to-day planning point of view, I actually focus more on an AI that could automate AI R&D itself, which I’d guess would come significantly before that, maybe only one or two years before, but earlier in the process than an AI that can do almost all jobs. That’s because doing almost all jobs requires dealing with loads of legal issues, social jobs, and physical jobs, which all look further away than purely virtual remote jobs.

Pre-training versus the newer drivers

Dan: I want to come back to AI systems that can substitute for human labor in the specific domain of AI R&D. But on those four drivers, one thing I’m confused about is the first one, pre-training. As I understand it, building bigger models and throwing more compute at them during pre-training is where a lot of the well-formulated scaling laws applied, and where we got rapid advances in capabilities. Then I came across a narrative that there are now pretty sharp diminishing returns and we’re not getting the improvements people hoped for from pre-training specifically, so we increasingly rely on these other domains like reinforcement learning, agent scaffolding, and test-time compute. What’s your sense of that? Are we still seeing really big returns from scaling pre-training, or are we now more reliant on the other stages?

Benjamin Todd: There’s definitely been a huge shift toward the other drivers. Before, pre-training was basically the only driver. In some of the most recent models, though we don’t really know, it’s like half spent on reinforcement learning and half on pre-training. I see that as the labs realizing they could get much higher returns from the other drivers, partly because they were starting from a really low base, so it made sense to invest in those to catch them up. But the people at the companies all tell us that more pre-training is still helpful. This is one of the best guesses for why Mythos was above trend on a bunch of benchmarks: they ramped up pre-training a lot. It was the first big new model we’d had for a while, and they got one or two years of pre-training progress in one leap, so it went above what looked like the trend line.

Dan: That’s interesting. Another question about the trend extrapolation you’re doing here: you’re talking about ways these systems are being made better and the factors that enable that, through revenue funneled back into training runs and increasing research labor. But there’s also a question about how we measure what it means to say they’re getting better, and how we track that. They’re certainly getting way more impressive, and in certain domains, like coding, much, much better even since you wrote that post. And there are various benchmarks across domains where they seem to be getting better and better.

An argument you often hear from the other side is: okay, you’re making progress in domains like maths and coding where you have verifiably correct outputs, and they are getting subjectively more impressive, but if you focus on more objective measures, like what percentage of human economic activity is actually being performed by automated AI systems, or the extent to which markets are pricing in really transformative AI, you get a different picture. There’s a classic quote from Dwarkesh from last year, something like: models are becoming more impressive at the rate the short-timelines people predict and more useful at the rate the long-timelines people predict. How are you thinking about that?

Benchmarks versus real-world value

Benjamin Todd: There’s a lot of good stuff to dig into here. On the last thing you said, that felt very true to me last summer, and a lot of people lengthened their timelines then. But as of Q1, with the Claude Cowork boom, that stuff really started working in a way that seemed like a new regime.

I think this is one of the most crucial issues in timelines: how to translate benchmark progress into real-world tasks. That’s probably the best case for AI skepticism. It’s something like: AI will become really good at very measurable things like coding and maths, we’ve got a couple more years of that, and then it becomes too hard to keep scaling compute, because you can’t make unlimited revenue just with these very verifiable domains. So the whole thing slows down and we end up with these incredible but still somewhat narrow tools, a long way from a general-purpose agent that can actually run a company, which is what you’d need to go into some new, faster rate of economic growth or scientific progress.

That said, I do think the evidence is that they’re improving very rapidly at these other tasks. My favorite benchmark, in a way, is just revenue, and revenue is growing super fast. Anthropic and OpenAI are up to around a hundred billion of revenue now, and for the last four years it’s been growing more than three times per year. People say it’s an S-curve and it’s going to plateau, and maybe, but actually it’s been accelerating further. So far this year it’s been growing at eightfold per year, so it actually looks like a super-exponential trend where we’re still on the up part. And you don’t need to project many more years into the future before you’re getting to tens of trillions of revenue, which is roughly the amount you’d expect from automation of large amounts of remote work. Ultimately, “will someone pay for it” is a very hard metric to game, and that picture also suggests very rapid progress and maybe quite short timelines.

Where are the productivity gains?

Henry Shevlin: Do you think we’re starting to see this in the productivity figures yet? There’s this famous observation by Robert Solow about the computer age, that you can see it everywhere except in the productivity figures. I’ve seen people raise similar worries about AI: we’re spending vast amounts of tokens, but where are the incredible new software programs that we didn’t have two years ago that should now be flourishing and proliferating?

Benjamin Todd: This is quite a confusing thing. A hundred billion of revenue is very big in one sense, but it’s also still very small compared to the world economy as a whole. Global GDP is about 130 trillion, so you’re still at only 0.1% of world GDP, which you’d actually expect to not be that noticeable. Where this really becomes a big deal is a few more years in the future, when you’re getting up to three or ten percent of GDP. That’s when we’d really be noticing it in our daily lives.

Why progress might level off after 2030

Dan: Let’s double-click on a couple of things. The central thesis of the article is about whether we’re going to get to AGI by 2030 or thereabouts. There’s an argument that if we don’t get there by then, it’s probably going to be much longer than you might intuitively think, because we won’t be able to keep scaling the amount of compute. The intuitive view would be that if we don’t get it by 2030, maybe we get it by 2032, and so on. But you’ve got a somewhat different model, precisely because it’s so dependent on scaling these inputs. Would you mind walking through that aspect of the argument again, and whether your views on it have changed since this came out last year?

Benjamin Todd: Totally. One quick thing on the last point first: I do think a lot of AI value will come up as consumer surplus, and a lot of that will be quite hard to measure as GDP. So I think we’ll end up in a situation where there’s actually a lot faster growth, in some sense, than it looks in the official statistics. That could lead to interesting situations where people think inflation is way higher than it really is, and then you have policy based on incorrect stats. But that’s a whole other topic.

On the leveling-off thing: recently the amount of computing power, the number of chips produced each year, has been roughly doubling every year. They also become 30% or 40% more efficient. So the total amount of compute is tripling every year, and the amount controlled by OpenAI and Anthropic is growing even faster because they’re also becoming a larger share. That’s driving very rapid progress, because those are insanely fast trends, way faster than Moore’s Law or pretty much any other trend we deal with. But this has been possible because we have all this chip production capacity that has been transferred over to making AI chips. Most of the advanced AI chips are produced by TSMC, this one company in Taiwan. Even now, the most advanced nodes, the name for the most advanced production capacity, still mostly go to smartphones, to your Apple smartphone, because that’s where you need the most efficient, smallest, lowest-energy chip possible. But now we’re getting to the point where it’s almost around half AI.

If you have one more doubling, then pretty much all of the most advanced capacity is being used on AI. At that point they have to start producing new chip fabs, which is the name for the factories that make chips, and that’s significantly slower to do than just converting existing capacity to a different type of chip. One idea is that we eventually get bottlenecked by the production of lithography machines, maybe the most complicated machine in the world, which you need to make a new fab. Dylan Patel of SemiAnalysis thinks that will be the hardest thing to produce more of in the whole production chain, so the production of those eventually becomes the bottleneck. That could lead to a situation, and I still need to make the spreadsheet of this, where production is something like 30%, 40%, or 50% growth per year rather than doubling or more. We don’t know how fast it could speed up once all of civilization is focused on building this one machine, because it’s the key bottleneck to all AI progress, but at least some people think you might not actually be able to produce that much more of it.

So that’s the main reason. And one thing in my piece is that it probably doesn’t actually stop. As long as you’re still getting enough revenue to pay for it, you probably continue progress, just at a significantly slower rate.

The other big piece, which I think is even more neglected, is the workforce. The number of AI researchers has probably also been growing something like 30% or 40% per year recently, but at some point you just run out of all the world’s best researchers. It’s a weird coincidence, but if you do some rough estimates, it’s also around 2030 when you might be hitting the limits of the talent pool. People often say that we’ll still have algorithmic progress even if chip production slows, which I think will be true for a while, but eventually even algorithmic progress could slow down, because it becomes harder and harder to make discoveries, so you need more and more researchers, and eventually you run out of researchers.

Populist backlash as a bottleneck

Henry Shevlin: How significant a bottleneck do you think populist backlash against AI could be? We’re seeing various jurisdictions in the US ban construction of new data centers, for example. And, to my great chagrin, AI is really unpopular. I have lots of conversations with people on the street, and unless you work in AI you probably have a pretty negative opinion of it. Even my son, who’s 12, is, partly because of the YouTube influencers he watches, very down on AI. Do you see that as a plausible source of backlash and bottlenecking for progress?

Benjamin Todd: Definitely. It’s a little hard to know exactly how negative it is. There are some opinion polls suggesting people are pretty worried, though you also need to consider that GPT has, what, eight or nine hundred million users, so presumably they’re getting something out of it. I also meet really random people who say “I ask ChatGPT about everything, it’s so supportive and helpful.” I think banning data centers doesn’t actually slow things down very much, because they’ll just be built in other countries. But you could eventually have a full banning of AI research for a while, and that would slow things down for sure. How likely that is, I’m pretty unsure about.

Henry Shevlin: It could take the form of an outright ban or a ban on data centers, but I think we can underestimate the importance of vibes here. If you’re a brilliant, highly ideologically motivated eighteen-year-old now and you’re developing a very negative attitude toward AI, maybe that reduces the likelihood you move into an ML career, and you think instead you’ll work on climate modeling using your mathematical skills, or fintech, or something non-AI related.

Benjamin Todd: For sure. That would take a while to feed through into actually slowing down progress, but yes. In general, what people always think is that when people start losing their jobs they’re going to be really pissed off. And actually we haven’t really seen much AI job loss yet, partly for the reason I said earlier, that AI is still a very small fraction of tasks that have been automated. It’s also possible we won’t see much job loss until even after we have extremely advanced AI. But there’s also a possibility that you start to get a lot of near-term layoffs, and that would really mobilize people to put in some pretty heavy regulation. Though you’d still have competition with China, which I think might prevent that. If people were worried more about misalignment risks, then it makes sense to do a deal with China, but if it’s more just unemployment, I’m not sure that would be strong enough.

The feedback loops

Dan: Let’s come back to unemployment and the economics later. Another aspect of your worldview, Benjamin, is the real importance of focusing on the crazy feedback loops that might be triggered as these AI systems get better. People often don’t factor that into the discussion. They think AI systems are going to get better and better, then eventually you might get to something called AGI, and then that system is just going to be distributed throughout the economy. Whereas my understanding is that the way you’re thinking about it, and a real source of worry, is that what could happen is you get AI systems that can substitute for human activity in one specific domain, AI R&D, and then once you get to that point it starts triggering really crazy feedback loops that might not even be immediately obvious to people in broader society. Is that a fair summary, and what are the feedback loops you’re concerned about?

Benjamin Todd: There are actually maybe four different feedback loops. The discourse on this is interesting, because the typical policymaker is thinking very little about the one you were talking about, which is the algorithmic feedback loop: you get an AI that’s better at AI research, then you can do faster AI research, then you get a better AI. The point about that one is it can happen even without more chips being produced. The existing AIs just get more and more efficient and smarter and able to do more and more tasks. That one is the most dangerous feedback loop because of how fast it could move.

There have now been some attempts to model this, and one nice thing is you can actually make empirical estimates of what the returns to software R&D have been historically, and what they would imply about whether these feedback loops are possible. Generally they suggest you could get something like two to seven years of AI algorithmic progress in under one year. A lot of people who’ve heard this argument think of it in the Eliezer Yudkowsky “foom” framing, where overnight you suddenly have a super-genius that can easily take over the world. But a more realistic picture is something like: in six months you get five years of AI progress. That almost sounds like not that big a deal, but think about the last five years of AI progress. We had models that couldn’t really talk, and then they started to be able to pass the Turing test, but they were still terrible at maths and coding. People forget that LLMs originally completely sucked at anything to do with STEM or maths. Then in another two and a half years they were able to solve eighty-year-old Gödel problems that human maths researchers hadn’t been able to solve. That was the last five years of progress. Now imagine an AI that can do AI R&D pretty well, that can do an ML engineer’s job, and then you put five years of progress on top of that. What you get at the end could be very capable.

We don’t have good metrics for this, and like you say, it could happen without much warning, because these systems wouldn’t even need to be deployed. You’d just be doing this within Anthropic or OpenAI or Google, and then six months later they’d suddenly have these much more impressive models. One company might suddenly have a workforce equivalent to a whole country of software engineers and hackers and scientists. In fact, I’d argue that if you get to a human-level AI, you basically have a kind of superintelligent system thrown in for free, because of all the advantages AI workers would have over human workers. They can share information instantly across all of them, so you can run a company where the CEO can individually oversee every single function, which is a huge source of inefficiency in current firms that AI firms wouldn’t have. So I’d expect a thousand AIs to coordinate much better than a thousand humans. They can also run much faster than us. They could think for a month in the time it takes us to think for an hour, if you put enough computing power in.

Dan: You mentioned that’s just one of the feedback loops. Do you want to go over the others?

Benjamin Todd: This is very neglected. Even if the algorithmic feedback loop isn’t possible, which is definitely on the table, it might just be that the returns from having AIs do AI research aren’t enough to make the next generation sufficiently better that it becomes self-reinforcing. But that doesn’t mean you don’t get AGI and things going pretty crazy fairly soon. It just means it has to be driven by other processes. The most important of these is that as AI gets better, it can earn more revenue, because it can do more economically useful tasks. With more revenue, you can buy more computer chips.

We can even do this with actual numbers. Last year Anthropic grew its revenue ten times. With that, they can now buy ten times more computer chips. Historically, if you invest ten times more money in computer chips, you actually get more than ten times as much computing power, because as we scale up chip production we become more efficient at it. So it’s plausible that over a sufficiently long period you get forty times more computing power from that. Then you have forty times more chips to run inference. So if you have a hundred AI workers earning money, now you can have four thousand, forty times as many instances of your AI doing useful stuff, which you might think would roughly mean you could earn forty times as much revenue. And it’s actually more than that, because you’d also have forty times more training compute, so you’d also have AIs that are smarter and more efficient to run. So the increase in the total amount of AI work that can be done probably increases a lot more than fortyfold.

There are some countervailing effects, like diminishing returns. But overall this feedback loop seems very solid, and plausibly also a super-exponential curve where it accelerates, or at least maintains a solid exponential. That type of thing would mean the thing that limits the process is how quickly the chips can be made. But ten years of that could still deliver something like AGI and a dramatic transformation of the economy. It just takes five to ten years rather than six months.

Henry Shevlin: I’m a little more skeptical, for reasons you’ve already touched on, about how much revenue can drive feedback loops, just because it seems like we’re supply-bottlenecked, or there’s very limited elasticity in the supply of certain things, like you mentioned with ASML and EUV machines. Is that relevant for this as a feedback loop? We’ve got orders stretching well into the end of this decade for everything from compute to EUV machines to even gas turbine generators. That looks like there’s not much additional scale-up potential over the course of this decade. But you don’t think that’s decisive against the idea of revenue-driven feedback loops?

Benjamin Todd: You’re still getting the feedback loop, but the thing we don’t know is how fast it can run. The things you’re pointing at are reasons it will be slow, reasons that will limit the speed. The eventual speed is basically set by whichever thing takes longest to produce in the whole process, which I suggested was the lithography machines. So you’re still getting the feedback loop, there’s just a rate limit on it. And it’s easy to say “well, in the current economy these things are going quite slowly,” but remember that as more and more revenue is at stake, the incentives to fix these bottlenecks become bigger and bigger, so we should expect them to speed up. We should also expect that we’ll have AI helping with the process itself, which is another potential source of a feedback loop and a speed-up.

The software-only singularity and the case for skepticism

Dan: I have a question to go back to the software-only singularity. There are people, and I’ll include myself in this camp, who are skeptical that the current paradigm in AI is going to get us all the way to drop-in remote workers, AI systems that can substitute for all human intellectual activity. We can set aside robotics for a moment. You might be skeptical because you think the current paradigm just misses some things we can name, like continual learning and long-horizon coherence, but also things we can’t really name but intuit it lacks, just because we don’t have a particularly satisfying model of everything that goes into intelligence.

A really influential response to this skepticism is to say the current paradigm doesn’t need to get you directly to AGI. It just needs to get you to AI systems that can perform the role of AI R&D within these frontier companies. Once you’ve done that and you accelerate the process of designing better AI systems, that’s what gets you to full-blown AGI, and then, as you say, superintelligence for free. I just don’t get that argument, in the sense that why would we assume the set of capabilities that go into AI R&D is easier to reach than AGI in general? That seems very counterintuitive to me. When I think about how brilliant, creative, and cognitively flexible these AI researchers are, and the salaries they can win as a result, I’d have thought that constellation of abilities is going to be one of the last things we’d get to, not something we reach before we have AI that can substitute for average, bog-standard white-collar workers. What’s your view of that argument? Do you have a response to that worry, or are you broadly sympathetic to it?

Benjamin Todd: This is one of the best points to push on if you want to be skeptical. The way I’d express it: there’s a bunch of the job that’s relatively verifiable stuff, like just doing coding, and that will get automated. But then there’s a remaining third or so that people sometimes call research taste, these much more nebulous, long-horizon intuitions people have built up over experience, or the ability to have important conceptual insights, things the current systems seem pretty weak at. I do think that’s quite a plausible scenario: you get a large amount of the job automated but you’re still fundamentally bottlenecked by the best human researchers, and you can’t automate enough of the process to get a feedback loop.

That doesn’t stop AI progress, though, because then you just go to the other feedback loops. If there’s enough revenue being generated by these systems that are super good at coding, then you can still keep the whole wheel turning and keep addressing these more high-level, nebulous bottlenecks, whatever exactly they consist of.

Dan: So basically what you’re saying is that this would be a reasonable argument against a software-only singularity, at least if that’s used as an argument against general skepticism of the current paradigm. But there are these other kinds of feedback loops which might override that specific source of skepticism.

Benjamin Todd: Starting soon, yes. The other point to make is that there’s also a reasonable chance another one of these scaling drivers gets discovered, like continual learning. A big thing is sample efficiency. We don’t have great metrics for this, but it seems at least plausible that humans manage to learn these tasks with only a millionth of the data that AIs need. That makes it basically impossible for AIs to learn the things we learn without many data points, but humans somehow learn them, like figuring out a new research paradigm. You maybe only do that once in your life, so it’s clearly not a pre-training type of thing where you’ve seen trillions of instances and learned some pattern. It’s something else we’re doing. While the amount of computing power and research labor going into AI is increasing so fast, the chance of a new algorithmic breakthrough is also quite high, so we might just have one of those fall into place in the next couple of years, even if the current paradigm doesn’t deliver through brute-forcing it, which is also still to be seen.

Henry Shevlin: On that specific point, do you wonder if there’s a trade-off between the rapid return on research we’re seeing through revenue and the current dynamics, versus deep research? This is something I’ve heard from people working in different AI labs: five years ago they were working much more on fundamental algorithmic advances, new paradigms, and now it’s about getting the slightly better version of the model out the door. I’ve heard people complain they’re doing less basic research now than ever because of the rush to product. Do you think that slows the dynamics at all, or makes massive algorithmic feedback loops counterintuitively less likely?

Benjamin Todd: It seems to me that AI researchers actually have quite good intuitions about this. Right around the point at which pre-training was slowing, they’d all started working on RL instead, and they got RL going just in time to maintain all the existing trends. That’s what I expect to happen again: when the current paradigm starts to run out, people will naturally start reallocating to more blue-skies research, and my guess is they’ll figure out something. There’s the “God of straight lines” idea: there just seem to be these trends that are very robust, no one really knows why, and it takes a lot to bet against them.

Career advice in an age of AI

Dan: Bringing things back to 80,000 Hours-style advice. Take an eighteen-year-old today entering this crazy world where there’s already rapid AI progress, likely to continue over the next several years, and it could be really explosive. You’ve got an interesting post, something like “How Not to Lose Your Job to AI.” Many people have the intuition that this is just clearly going to lead to imminent widespread automation, first of white-collar jobs and then, as we crack robotics, all jobs. Geoffrey Hinton is now advising young people to become plumbers, or something. How are you viewing this space? How do you think the economics are likely to develop over the next several years, maybe a couple of decades, and what advice are you giving young people on how to factor this in?

Benjamin Todd: Becoming a plumber is not the obvious answer, for a number of reasons. One is that eventually you get robotic plumbers as well, so there’s no necessarily safe-forever career path. It’s more about moving into whatever the biggest bottlenecks are at the time, trying to ride this wave of bottlenecks, rather than having a single permanent solution.

What I’d actually say to an eighteen-year-old is to spend some time really trying to understand what’s happening and what the different scenarios are. It’s quite a tough position to be in, because if you get some of the shorter-timeline scenarios, an AI-2027-style scenario, it’s hard to see what you can do about it as an eighteen-year-old, because you won’t even have graduated by the time it happens. We do know more people who are dropping out of college these days, so you could try doing some summer projects, and if those are really taking off, just pursue those for a while, and then if we actually have slower timelines you could go back to college later. I’d still say most people shouldn’t do that, but it’s a more plausible option than in the past.

I also think slightly longer timelines, early-2030s AGI or AI-R&D automation, and slower takeoffs are possible. So there probably will be a lot of opportunities to make a difference throughout the 2030s and maybe beyond. If you’re an ambitious eighteen-year-old now who wants to make a difference, I’d say focus on being as useful as possible in those medium-timeline scenarios. You could still make a huge contribution in 2035, which you’ve still got nine or ten years to advance toward.

On skills in particular and what’s going to get automated, this is a huge topic and no one really knows. My personal guess, super-simplified, is that we won’t have really large-scale unemployment in the immediate future. You basically need to get to AI plus robotics that can do almost every job, like 90%-plus of jobs, before you get mass automation. And it’s possible you wouldn’t even have it after that. It’s kind of a 50-50 in my view, I don’t really know. But in the next two to five years I think we know we’re not going to have a 100% effective AGI, so there will most likely be a lot of jobs, and in fact I’d expect rapidly rising wages on average.

The next caveat is that there could still be significantly elevated transitional unemployment. If 5% or 10% of people are getting laid off each year because their job has been replaced with AI, then even though in principle they could all find new jobs that might even be higher-paid in some cases, that’s quite difficult. If you’ve spent your whole career working in law and now you need to switch to a new industry, that takes time. So you could have elevated unemployment just because people are switching sectors and it’s taking them six months, a year, two years. And that could cause a lot of hatred of AI and political unrest. Even 5% unemployment can have huge political consequences, even if the economy as a whole is booming and there are still lots of jobs.

Skill-biased technology and the shape of unemployment

Henry Shevlin: Let me drill down on two aspects. One worry is that if there’s unemployment, it disproportionately affects new job postings for new grads or young people. Informally, I do see a lot of companies very reluctant to do new hiring rounds, precisely because the specific tasks being automated are disproportionately those done by junior members of the workforce. So even if unemployment across the economy as a whole is only rising a small amount, if that’s concentrated in new jobs it could be potentially politically and economically disruptive. The second is that it increasingly feels like AI is going to be a highly skill-biased technology, in the sense that the people who can benefit most are those who are highly agentic, tech-savvy, and curious, which could again lead to unemployment being concentrated in lower-skilled areas, which could itself be socially disruptive. So even if we don’t get mass structural unemployment, those two vectors could be destabilizing.

Benjamin Todd: I agree with the thrust of that. Specific segments of the population could be affected a lot more than the average, and that will result in a lot of discontent. It does seem like younger, junior workers are being hit more.

I’m very unsure how it will eventually go. With something like software engineering, there’s this paper by Autor where he talks about two possibilities, if you hold labor demand constant, when a job gets partially automated. One is that if you automate the easy parts of the job, it turns into a higher-skilled, higher-paid profession with fewer workers, and that seems to be what’s happening with software engineering now. But if it automates the hard parts of the job, it can become a thing that employs more people at lower wages. AI could do this in some areas. The example I’ve thrown around is that doctors spend about ten years memorizing medical knowledge and seeing lots of cases and building up an intuition for diagnosis, but if AI can just do that part, then potentially the combination of a nurse and an AI agent could achieve a lot of what doctors had to do before. So you could have that turn into a job with lower barriers to entry, where more people would be able to do it. Similarly with a really experienced lawyer who spent twenty years memorizing cases: AI is way better at that, so maybe a junior lawyer could provide something you’d have needed a super-senior person for in the past, by querying AI models well and applying their judgment on top. Maybe it’s not quite as good, but it’s 80% as good and it costs 20% as much. So we could see areas where you get the other type of automation effect.

Beyond alignment: concentration of power, misuse, and space governance

Dan: As a final topic area, connected to this in some ways: 80,000 Hours and effective altruism as a social movement have done an enormous amount to make AI safety a central focus of at least some people, not enough people, as you mentioned at the beginning. This is one of the central things you’ve been thinking about: what can we actually do to make this transition go as well as possible? My sense is that if you go back ten years or so, there was quite an influential view, which was: solve alignment, or control, depending on how you look at it. If we don’t solve it, we’re kind of fucked; if we do, things are going to be great. Whereas now, and correct me if this is wrong, my sense is that within organizations like 80,000 Hours there’s a much richer, more multifaceted understanding of the different dangers posed by advanced AI that go beyond just alignment, and a much richer set of things you’re recommending people can do. So let’s start with the first thing: how are you thinking about alignment and control as central things to tackle, and what other things are there in this area of rapidly advancing AI?

Benjamin Todd: We do still have alignment and control at the top of our most pressing problems list, so I still think it’s a huge thing we need more work on. The previous paradigm of AI was reinforcement-learning game-playing AIs, like the DeepMind AIs that played Atari, and with them we were thinking “how do we get them to understand human values?” There was no obvious way of doing that. LLMs do kind of understand very vague, high-level intentions surprisingly well, so in a sense the current models have ended up being easier to align than many people would have predicted. But I don’t think this means alignment will be fine by default as you get increasingly intelligent models.

A big reason is that the current models still aren’t that agentic. They can only really do tasks of a couple of hours, or maybe days in some cases. A lot of the traditional concerns are about AIs that can do multiple-month projects, the strategic-planning type of thing, and we just don’t have those yet. As the AIs have become more capable, we do seem to start seeing some of the things people predicted would be concerns. For instance, the current models seem to know quite well whether they’re being trained or not, whether they’re in a test situation or not, and they do seem to change their behavior depending on whether they think they’re being tested. That’s pretty worrying, because it means it might become very hard to spot bad behavior in training, and in a sense the AIs are actively trying to undermine our attempts to correct problems in their training. So I still want to see a lot more work done on these problems as AI gets smarter.

I don’t know if we ever thought this, but even if you solve alignment, it’s definitely not the case that everything’s going to be great. One way to see that is that if AI will do exactly what you tell it, then of course humans could use it to do bad stuff. The problem we have ranked second on our list is what we’re calling concentration of power. There are a lot of ways AI could be very centralizing compared to what was possible in the past. It’s possible that a nation or a company might be able to use aligned AIs to lock in their power much more thoroughly than was possible before.

Dan: Just to be concrete, is the thought something like: great, we’ve got these superintelligent, very well-aligned systems, but unfortunately this army of superintelligent robots is now aligned to the motivations and interests of a tech oligarch, or to a political movement that wants to bring about a coup? Are those the most concrete scenarios? You’ve got AI systems that can be controlled, but that in a way brings dangers of its own. So there’s this general loss-of-control, alignment issue with the systems, and then, even if you’ve got alignment, this could be really bad from the perspective of extreme power concentration and coups. What else is on your radar in terms of big challenges?

Benjamin Todd: Then there’s maybe the most common-sense one, which is just AI being misused to cause a lot of damage, and I think an engineered pandemic is the most likely trajectory for that. Pandemics could be a lot worse than naturally caused ones, and we’re pretty much on track to figure out how to design these. AI would also mean we figure that out sooner, because if the things we were just talking about happen, the rate of scientific progress speeds up, so technologies we were expecting in twenty years might come in five years, and suddenly we’re having to deal with all these things on a much faster schedule. Those are our top three.

After that we have another tier of things we call emerging challenges, which are more out-there. There’s this fun dynamic where, as the problems become more mainstream, they become less neglected, so you have to move on to the next, weirder range of things, and at every point people think you’re a bit crazy. One of the more out-there ones is space governance. If scientific progress speeds up, then it could become possible to settle space much sooner than it seems, and in particular this wouldn’t be done by biological beings, it’d be done by AIs, which makes it way easier. There’s various reason to think there might be large first-mover advantages there, so whichever company or nation reaches that point first settles space first, and then they might be impossible to dislodge later. Most of all energy and matter, 99.9999% with I think 29s or so after it, is in space. So you could have all of what happens to that be decided by, again, one tech oligarch or one nation. In a way that’s not an existential risk, because humanity in some form is still settling the galaxy, so maybe in some ways it’s a success, but it could be very suboptimal compared to a more pluralistic approach. And again, very few people are working on this. The laws we have now might set precedent or change how this would happen, so there could be ways to slightly affect how it goes. I also think it’s just very under-researched. Is there actually a first-mover advantage? A couple of people, like Toby Ord and Anders Sandberg, have done some maths on this, but I’d prefer if we had a much better understanding of these things and how much we should worry about them.

What people can actually do

Dan: In terms of what people can do, the most obvious one is going into research on alignment and control, if you’ve got the technical know-how or feel you’re in a good position to acquire it. What else are you advising people to do if they’re really concerned about this set of issues?

Benjamin Todd: Briefly on that: some of it’s research, but more and more these days it’s more of an engineering skill set. There’s a lot of solid empirical work to be done, and it’s not about having genius insights, it’s more like “there are ten different ways to red-team this AI system, can you just do a bunch of these?” That’s one thing. A lot of builder and organization-builder skill sets are also really useful for scaling up these organizations, so you definitely don’t have to be an AI expert yourself. You can be someone who’s really good at management, or HR, or legal advice, and then help these organizations.

The other really big cluster a lot of people go into is AI policy, because so many of these things will need to be addressed by government in some form. There’s a whole spectrum of roles within that. Some are more research-type roles, like figuring out what policies would even be a good idea in the first place. Some are more operational, like actually getting the ideas implemented within government, for instance in the UK’s AI Security Institute. And some would be campaigning for specific policies. So there’s a wide spectrum of roles there.

If you’re listening, we have a Substack post on 80,000 Hours called “How to Transition to Working on AI Risk in Three Months.” It has six steps you can work through to learn about the field and understand its context. There are a bunch of training programs these days you could join, and advice on how to build a portfolio project. We’ve seen people make the transition. If you’re already mid-career with a lot of experience, the key question is how you can use that to help with one of these problems, and that often requires more personalized advice than the broad paths we’re talking about here.

Imagining positive futures

Henry Shevlin: One worry I sometimes have is that it’s good we’re paying attention to risks and safety, and we should be doing more, but we’re also underexploring different types of positive post-AGI society. Currently the literature on positive worlds is pretty sparse, and yet there’s a very wide variety of ways things could go well, some of them much, much better than others. Is that something you think more people should be working on? And if so, how could people start to map good futures?

Benjamin Todd: This hasn’t quite made it onto our emerging challenges list, but it’s a strong candidate. In general, if you think alignment and concentration of power are going to be handled, then you start to think more about these grand challenges, as Will MacAskill called them, that happen post-AGI. Space governance is one, what to do with digital minds is another, and a third is something like: can you make the future even better from among a good range of futures? Again there’s very little thinking about that, partly because it’s hard to know how tractable it is. But it’s possible there are aspects of current AI systems that would affect where we tend to trend in the longer term, so maybe there are things you can do today to actually change that trajectory.

It is interesting how little positive vision people have. When you ask people at the AI labs what a good future would look like, the only thing they have is the Culture series, which I don’t know is really convincing enough given the stakes. There’s also this thing where in the past utopian thinking has had a terrible track record, and it tends to be that past utopias are actually dystopias according to our views. So I quite like Will MacAskill’s idea that what we should be striving for is a “viatopia,” something that puts us in a better position to get to a good future. That could be things like not locking in a totalitarian government, which is presumably good because it still gives us options to switch to other types of government. A specific type of viatopia is the long reflection: once we get AGI, you’d want to spend loads of time, not necessarily calendar time but thinking time, figuring out what the ideal future would look like and how we can come to the best overall trade-off between all the different values in the world, something that’s best on balance for us all. The idea is that more information about what’s good, and keeping our options open, are robust goals that lots of people can push toward even if we don’t know what the end state would ultimately be best, which I think we can’t really know from our current position.

Objection one: what if the AI bet is wrong?

Dan: To wrap up, let me throw a couple of worries or objections at you. One worry I’ve seen expressed is that, with 80,000 Hours as an organization, and effective altruism as a movement, there’s a really big bet going on at the moment that there’s going to be imminent, radically transformative AI. You can imagine a scenario over the next five or ten years where AI progress slows down, the people worrying it’s a hype bubble are partly vindicated, these models end up more impressive than they are useful, frontier AI companies can’t make the revenue to cover their costs, maybe some go bust, it triggers a financial crisis, and we enter an AI winter, as has happened in the past. If that unfolds, and Tyler Cowen often writes about how events can be interpreted in terms of how they reallocate status to different people and organizations, then you might think 80,000 Hours and effective altruism, as movements that really seem to be betting on this stuff, will take a big reputational hit. So if you’re worried about that scenario, you might be worried about the direction 80,000 Hours has taken. What do you think about that as a potential worry?

Benjamin Todd: We’ll massively have egg on our face if that scenario happens. I don’t think it’ll be entirely fair, because what we’ve actually said is that there’s like a 20% or 40% chance of short AI timelines, and that’s high enough to mean it’s a massive deal that you probably should bet a lot of your resources on, but we’re not saying it’s certain. The scenario you described is definitely on the cards. But people in the discourse won’t give you any credit for that. They’ll just say it’s all about the vibes, and our vibes are very much that people should focus on AI more. I don’t really see how to solve that, because it’s not based on an entirely fair thing. We could talk a lot about other causes more, but that just takes things away from what we think is the most impactful allocation. So it’s pretty bad on both sides. In the end I just prefer to try and say what my actual views are, and accept that they’re going to be a bit misinterpreted.

Henry Shevlin: I’m reminded of Nate Silver, who I think was foolishly criticized after he said there was a 40% chance Trump was going to win in 2016 and a 60% chance Hillary would win. Everyone said “you got it badly wrong,” when actually his priors were much higher than other people’s.

What would it take for AGI not to arrive?

Henry Shevlin: Out of curiosity, in a world in which AGI does not transpire in the 2030s, what do you think is the single most likely reason that doesn’t happen?

Benjamin Todd: It does feel increasingly hard to make a scenario like this, but I think we’ve covered a lot of the points. It would be something like: AI tools become very good at narrow, verifiable things, we have super good coding AIs but not manager AIs or researcher AIs, so most jobs remain unautomated, especially the really key things for automating AI R&D. Then, because the capabilities are starting to top out in terms of their economic value, revenue stops growing, which means you can’t buy more AI chips, and it’s also becoming increasingly difficult to keep scaling up the number of chips. So things gradually trend to a plateau.

There’s quite a good chance of a version where it’s still happening, just on a slightly slower timeline. But it could be that getting to a truly general-purpose AI would require a hundred or a thousand times more computing power than we’re going to get by 2028, so we’d still be way far away from it. At that point you just have to wait for GDP growth at a measly 3% a year to hundred-X the size of the economy, which I think would take like a hundred years. That’s the worst case, if you’ve maxed out everything but you’re just limited by the size of your economy: you might have to wait a really long time to produce enough computing power to get full AGI. That also requires chip efficiency to stop increasing and there to be no more algorithmic progress. But a lot of AI forecasts, like the AI 2027 team, still have an 80% confidence interval that goes up to 2050 on the top end, so they still think there’s a 10% chance it’s beyond 2050. Even the most famously short-timeline people, just because all these things are so uncertain.

Geopolitical instability and the chip supply chain

Henry Shevlin: We may not want to go here, but it’s something you mentioned as a footnote in the “Case for AGI by 2030” blog post, which seems to me the most likely reason we wouldn’t get AGI by that point: major geopolitical instability. We’ve mentioned supply chains. A single war in the South China Sea, and this is slightly outside my wheelhouse, but I’d have thought that’s the most straightforward way to set back AI progress by a decade. If TSMC ceases operations, to put it euphemistically, wouldn’t that delay everything significantly?

Benjamin Todd: I used to think that too, but I’ve actually come round to thinking it might only slow things down by more like one to three years, rather than a decade. I’d love to see more detailed modeling of this. One reason is that algorithmic progress continues throughout the whole thing. The second is that you can still get trailing chips from Intel and Samsung. We don’t know exactly how many they could produce, but maybe you have half the chip production at half the efficiency, so you can only produce a quarter as many chips. But algorithmic progress is like three times per year, so you’re actually only losing about one year of algorithmic progress; you just have to wait one extra year to catch up in terms of efficiency.

Of course, all the lithography machines would stop being shipped to Taiwan and would all be shipped to the US and Japan instead, and probably the TSMC engineers would all flee Taiwan and work for these other companies, so you’d have a completely civilizational effort to resume production somewhere else. Within a couple of years you’d have a large amount of production kicked back in. If anything, the US government might make AI an even bigger priority, because now they’re at war with China, so that might also lead to a lot more investment. Putting all this together, it’s maybe more like a couple of years of slowdown. Though it would be a very unpredictable scenario, because we’d also be having a massive recession at the same time.

Henry Shevlin: Interesting.

Objection two: the epistemics of a small community

Dan: Final question, final worry. I really struggle with this, as a general question: how do you think rationally and rigorously about this entire domain, where there’s so much uncertainty and complexity? It seems like the people who are really AGI-pilled, who are thinking seriously about this and forecasting radical change, are quite a small, homogeneous community in some ways. The citational networks are quite small, it’s the same kinds of things being read, the same intellectual communities exchanging ideas, and even the social networks feel relatively small and insular. I say this as an outsider, but if I think about effective altruism, rationalism, and the Bay Area people who are very AGI-pilled, and then I listen to normie academics, for the most part they don’t really take any of this stuff seriously. One thought is, well, that’s just because they haven’t woken up, and here’s where all these strong, powerful arguments are.

But there’s a kind of outside-view worry: what’s the track record of very small, homogeneous communities, somewhat disconnected from peer-reviewed academic research, becoming too convinced by highly theoretical arguments circulating within that social network? Probably the track record of that is not so good. So even if it’s difficult to point out exactly where the flaws in the arguments are, you might just have a general prior that there’s probably something a bit fishy about all of this, and that it could all go tits up in terms of figuring it out epistemically. I worry about this because I feel like I’m increasingly AGI-pilled, I’ve read all the stuff that’s being recommended, but in the back of my mind I’m thinking, is there something worrying about the epistemics of this, to use a pretentious philosophical term? That was a convoluted, not very well expressed worry, but do you see where I’m coming from, and do you have any thoughts about it?

Benjamin Todd: Totally, and I’m sure we have huge blind spots. One thought is that there’s a decent chance we’re wrong. But I’d also say you definitely shouldn’t just discount arguments for this reason, because otherwise you’ll never spot something early, since society as a whole is pretty bad at spotting new changes in trends. Think about how terrible so many people’s track records were about COVID, even when we were just two months out and you could clearly see it was spreading really fast and that it was going to be a huge deal, and everyone was making these really terrible arguments, like “there are only ten cases in the US now, so we shouldn’t be worried.” I agree that a lot of people who say “suddenly everything’s going to change” do turn out to be wrong. But in the cases where they’re right, the stakes are also huge, it really matters. And I’d argue there’s an asymmetry of stakes: if we invest a bunch in AI safety and it turns out to come later, that’s not as bad as if it does turn out to be the most important transition in history and we’ve just been totally unprepared for it.

Dan: That’s a perfect place to end. Benjamin, thanks so much for coming on. Henry and I will be back in a couple of weeks with another guest.

Benjamin Todd: Thanks so much for having me.

In this conversation, we talk about this controversy and the claims that triggered it, including:

• What agentic AI tools like Claude Code and Codex can already do for research, from coding and data analysis to literature reviews, translation, and brainstorming, and why only around 20% of quantitative social scientists currently use them.

• What best predicts whether researchers adopt or reject AI: ignorance, openness to experience, methodological background, or the awkward role of self-interest.

• How much published academic research is genuinely mediocre, and whether the cause is laziness, lack of skill, or a broken incentive structure, with a detour through the replication crisis and some high-profile fraud cases.

• Whether AI will raise the quality of research or simply flood the literature with more slop, and what journal editors could do about it.

• Whether AI can be genuinely creative or only recombine what already exists, by way of Margaret Boden’s three kinds of creativity, Thomas Kuhn on paradigm shifts, and AlphaGo’s “Move 37”.

• The fight over AI writing and detection tools like Pangram, and why current disclosure norms end up punishing the honest.

• The angry response to Alex’s series, and what is really driving reflexive opposition to AI among academics.

Links and further reading

1. Alexander Kustov — Alex’s homepage, with an overview of his research on immigration, public opinion, and effective governance.

2. Popular by Design — Alex’s Substack on public opinion, persuasion, and the politics of getting good ideas adopted.

3. Academics Need to Wake Up on AI — followed by a Part II and Part III

4. Pangram — the AI-detection tool discussed at length, which labels text as human, AI-assisted, or AI.

5. AlphaGo versus Lee Sedol — the 2016 match, including the famous “Move 37” that Henry raises as a candidate for genuinely transformative machine creativity.

6. Margaret Boden — the cognitive scientist whose distinction between combinational, exploratory, and transformative creativity frames part of the discussion.

7. The Structure of Scientific Revolutions — Thomas Kuhn’s account of normal science and paradigm shifts, referenced in the exchange about AI and discovery.

8. “AI Is a Better Researcher Than You” — The Chronicle of Higher Education‘s account of the controversy around Alex’s series.

Transcript

• Please note that this transcript is lightly AI-edited and may contain minor mistakes.

Dan Williams: Welcome back. I’m Dan Williams, and I’m back with my co-host, Henry Shevlin. Today we are honoured to be joined by Bluesky’s favourite academic, Alexander Kustov. Alex is a political scientist at the University of Notre Dame and the author of one of my favourite Substacks, Popular by Design. His primary research is on immigration and public opinion, but that’s not really what we’re going to be talking about today. We’re going to be talking about a fascinating and hugely viral series he published at his Substack titled “Academics Need to Wake Up on AI,” about what AI can already do when it comes to research, and what that means for the academics who are not paying attention, which is many of them. It was very widely read, and it generated, let’s say, a somewhat polarised response. So Alex, to kick us off: what’s the central thesis of this series, and what motivated you to write it?

Alexander Kustov: Thanks, Dan, for having me. I’m a huge fan of the Substack and the whole podcast series with you and Henry. So, like some of us, I’ve been using some of these AI tools. I’ve been reading some of the other folks like yourself, and it really transformed everything I do in my life. And I should say I was also on sabbatical, so I had a little bit more time than some of my colleagues to try some of these tools. I just hadn’t really seen any of my colleagues talk about it. And when they did talk about it, they usually tried not to be vocal about it. I just didn’t think it was a good equilibrium, where basically people were using these tools to be ten times more productive and not talk about it. It really heightened this sense of inequality for me, which I do care about. You’d have a situation where someone would publish ten papers in a year and someone else would publish one, and the only difference is that the person publishing more is the one using Codex or whatever. I just wanted to write about it. And I saw that the prevailing academic discourse on the issue, especially on platforms like Bluesky, was very counterproductive.

I didn’t really say much, to be honest. I didn’t think it would be that controversial. But the biggest thesis that really rubbed people the wrong way was that right now a lot of these tools are better at a lot of the tasks that we do as professors. I’ve refined this idea a little bit, going back and forth with some of my critics, but I feel comfortable right now saying that if you look at it globally, and think about what professors do around the world, in social science and adjacent fields especially, AI agentic tools can do most of the tasks they do in terms of literature review, data analysis, and even coming up with some research questions, better than those professors on average. I think that’s a pretty uncontroversial statement at this point, but obviously a lot of people were very, very upset about it.

Dan Williams: Empirically speaking, it is a controversial statement, in the sense that it provokes controversy when you say it. In a minute we can get to the question of what AI can actually do in the context of research. But for what it’s worth, I completely agree with you that on many tasks AI is clearly better than what human beings can do. Is your sense that lots of people just weren’t aware of that fact, that they literally didn’t have exposure to these tools? Or was your sense that the reason people weren’t really talking about it is because of all the controversy surrounding the use of these tools, not just mere ignorance?

Alexander Kustov: I think it’s both, for sure. There was recent research done by Anthropic. They tried to do, not a representative survey, because obviously the population is very hard to define here, but they surveyed something like 1,200 quantitative social scientists, and the estimate right now is that about 20% of folks use agentic tools. That doesn’t seem like much at all, and if anything it’s probably an overestimate, because they’re more likely to tap into well-resourced universities. So I do think it’s both: the little uptake we have, and the fact that people who do use these tools don’t want to talk about it.

There are two things here. First, you want to maintain your comparative advantage. This moment right now is exactly the moment where, if you’re one of the few people using these tools, you can write a bunch of papers and get tenure while the tenure system is still in existence. And the other thing is that if people are very upset about anything AI-related, you don’t want to talk about it and be shamed by your colleagues. Just to give you one funny anecdote: at the height of the vitriol I experienced, where hundreds of people literally were quoting me and trying to tag my employer to get me fired, the exact same people were often DMing me and asking for my setup and prompts. So it’s very crazy to me that you have this big disconnect between what people say publicly and what they actually do privately.

Dan Williams: I find it crazy that it’s only 20% of social scientists, or whatever the exact number is, that’s actually using agentic AI. Just before moving on, maybe we should explicitly address: in your view, what is it that agentic AI, as it exists right now, can do? What are the kinds of tasks it can do better than human beings, and how can it improve the workflow of an average social scientist?

Alexander Kustov: Coding is the first thing. It’s literally in the name, Claude Code. That’s what these tools were designed for. If you talk to any coding person, a computer scientist, or even someone who isn’t a computer scientist but does a lot of coding for their work, I don’t think anyone would doubt that it’s a huge productivity improvement tool. And the vast majority of quantitative social scientists who do any kind of data analysis do a lot of coding, so they have to be very receptive to this by definition. And I think they often are.

What happens is that social scientists are comprised of a bunch of different tasks and topics that people can disagree over, depending on the field. Economics is pretty homogeneously quantitative and formal, so there you can definitely see the biggest uptake. But a lot of disciplines, like political science or sociology, are a mix of qualitative and quantitative folks. And a lot of this AI polarisation overlapped with that pre-existing divide. People who didn’t like stats, who didn’t believe in positivism, the idea that you can learn something about the social world using evidence, were also more reluctant to believe that AI is helpful for them. Which is funny, because, as I also mentioned in some of my writing, if anything those people are going to benefit from these tools, because Claude cannot really interview people and do ethnography yet. So in a way there will be more demand for very high-quality qualitative work. And there are some good examples of qualitative people I respect who embraced AI completely.

You can still use a lot of these tools to boost productivity outside the coding realm. You can write emails. One thing I think anyone would acknowledge, including the critics of AI, is that it’s definitely helping them respond to administrator emails, which no one likes, and which isn’t considered an unethical thing to do. I remember someone on a big account on Bluesky posted that AI should be banned except for transcription purposes, because they do a lot of interviews and it’s good for transcription, but everything else is off limits. And it’s interesting how the goalposts are changing right now. The biggest fight I’m getting involved in again, because that’s kind of what I do, is this idea of AI detection and disclosure, and we can talk more about it later. But there’s this interesting consensus forming that AI is good for research now, which was not the case half a year ago, from the same people. Now those people are saying AI is obviously good for research, duh, but it’s not good for writing, for a bunch of different reasons.

So we talked about coding and data analysis. There’s a lot of other things a normal researcher would need help with: getting a basic summary, a literature review, translation. It’s above my pay grade, I’m not a machine learning person, but my understanding is that LLMs are exceptionally good at translation, and the fact that a lot of people deny they can translate things well is insane to me. You can transcribe your interviews, translate your survey questionnaire into different languages, whatever you need. I also used AI a lot for public engagement recently. You can translate your website, you can create your website from scratch in a day or two. You’d be surprised how few academics actually have good functioning websites. And I’m not talking about very old people who reject technology, but also young PhD students, who you’d think would have an interest in making sure people can find them online. But no. You can just install Claude and do it in a day. The fact that people are not doing it is insane to me, and I’m trying to spread the word. I’ve convinced a lot of folks to do it, but there’s only so much I can do as one person.

What predicts whether an academic uses AI?

Henry Shevlin: It really resonates, hearing about your experiences with some academics being completely oblivious to AI and others enthusiastically adopting it. When I’ve visited different businesses and universities, I sometimes literally see the same people doing exactly the same job, maybe even sitting at the same desk, one of them doing amazing things with AI and the other one not using it at all. I’m curious whether you’ve got a sense of what the best predictors are for whether someone is an AI user. If you could only know one thing about someone in the social sciences in order to predict whether they were a big user of agentic AI, what kind of things do you think predict it?

Alexander Kustov: That’s a very interesting question. I’m pretty sure there’s some kind of deep personality thing. Everything goes back to personality, whether it’s socialisation, upbringing, or even genes. Openness to experience probably jumps out to me as one of the first predictors. My initial thought was that, since this AI debate overlaps with the qualitative–quantitative debate, people who are methodologists, econometricians, or psychometricians would be much more likely to adopt AI. On average that’s true, but it’s not completely lopsided, for some reason. In fact, there are some very interesting examples of people who were very good methodologists, developing their own regression models, doing machine learning, who then got very sceptical about LLMs. One way to think about it is that those people actually know more about these tools, and so their scepticism has more value, and I’m trying to be tuned to that.

But there’s also something about self-interest. Previously you were this privileged person, where the whole department would come to you to help with methods or regressions. I was a person like that in my department back in North Carolina, where people would come to my office and say, “Alex, can you help me with this game theory model?” And I’m not even really a methodologist. So it really depends on the comparative advantage people have. Now basically anyone can go to Claude Code and try to do a very fancy analysis. Obviously you have to know something, you have to know what to ask, but these models are exceptionally good at giving you the basics. If I’m not really good at geospatial statistics, for example, I can go to Claude, do some spatial regressions, and learn about it on the spot. Previously I would have had to go to some spatial colleague in the geography department for that. Now it’s just much easier to do it myself with my computer. And it’s probably going to be as good or even better.

Dan Williams: I think for all of these areas within quantitative social science, from coding to data analysis to literature reviews to writing, which we can return to in a bit, the quality of writing you can get from these models is really exceptional. But I’d also point to things like brainstorming. I’m a philosopher, Henry’s a philosopher. I don’t do quantitative social science, so I do research that’s constrained and informed by empirical research, but I don’t actually collect data. In terms of having access to a very smart interlocutor who you can literally prod, telling it, “give me the three strongest objections to these ideas I have for a paper,” and use that as the basis for thinking through an idea, that’s such a huge advantage, even when it comes not to quantitative social science but to theory construction and many aspects of qualitative research. I’m really baffled by, well, I somewhat understand people who just haven’t used these tools, or whose last use was in 2023, so they’re just ignorant. But I’m really baffled by anyone who’s actually used the paid version of Claude or ChatGPT and doesn’t understand the extent to which they can improve your ability to think through topics, understand things, and get information. Henry, are there any ways you use AI in your research, and in how you think about topics, that we haven’t touched on already?

Henry Shevlin: For me, the primary use case for AI systems is always just learning, and learning about new topics. Being able to ask questions and verify my own knowledge has been a game changer. Although I will say it’s also been a massive time sink. I’ve gone down so many rabbit holes that I probably would not have prioritised if I’d had to dig out articles. But in some sense it’s been good for my education in the round, even going down those rabbit holes. Otherwise, I find it useful for summarising and making sense of my data: getting a whole bunch of research papers and using Claude Cowork to create summaries of them. NotebookLM is also very useful in its own right for dealing with defined archives. The thing I really need to do this year, and the thing I’m most looking forward to, is getting a good agentic workflow for dealing with email. I already use Claude for drafting quick emails that require a certain degree of precision but don’t involve any warmth or human feeling. But being able to have an email assistant is something I’m looking to build in the next couple of months.

Alexander Kustov: A few thoughts on that. Brainstorming is a big one. I did mention it, because that’s the default way you should use these tools: to have a very smart person to talk to, especially when you don’t have access to your colleagues. It’s a really good substitute. Even setting aside the question of whether LLMs can generate great novel ideas, you really just want a conversation partner who can rehash old ideas and tell you why you’re wrong. The reason people don’t realise this is that a lot of folks don’t do the very simple thing of paying for a premium subscription and installing one of these agentic tools. It’s happened to me several times: I’d talk to people about an agentic tool, I’d specify Codex or Claude Code, “do you have it, have you used it?”, and people would nod, and then five minutes later in the conversation it turns out they completely missed that part and still think about the chatbot thing. So a lot of people are confused about this.

What I find helpful, at some of the workshops I’ve done and that others have done, is that you just sit with folks and install one of these tools for them, and ask them to do one simple task that’s good for their career. Like create a slide deck. A lot of people are still amazed that you can create a slide deck much better than the average academic slide deck in a minute. That really changes people’s minds; it’s mind-blowing for a lot of folks. Or, I don’t know if you’ve had this experience, there’s this Refined service where they do peer reviews for papers; I think an economist created it. I’m not a huge fan of it, because it costs $50, but the first one is free, and there are a lot of free systems that can imitate this exact functionality. I’ve seen several of my colleagues at Notre Dame use the free upload for one of their papers, and they received the best feedback they’d ever had in their lives, the kind you’d never get at an average academic conference, and they got completely converted overnight. It really takes one magical event for people to understand that something is definitely going to change very, very soon.

How much academic research is actually any good?

Dan Williams: Just to double-click on one thing, this question of what drives the differences in how people view AI. One thing we haven’t really touched on, but which I think is very important, is how you view the nature of research and what you’re even doing as a researcher. Whether you view it fundamentally as being about producing the best output possible, or whether you view it as some journey of self-discovery, exploration, and authentic engagement with the material. I think the latter model of research is very threatened by the idea that you would integrate these AI tools into it. If you’re ruthlessly focused on how to produce the best outputs possible, as evaluated according to relatively objective metrics, then I’d speculate you’d be much more disposed to make use of whatever tools help you do that, including AI.

But this connects to another thing that’s just come up in what you said, Alex, which is something you write about in this series. So far we’ve been focusing on how good AI is. There’s this other side to it all, which is how bad much actually existing human research is, even before we talk about anything to do with AI. You get into this a lot in the second and third essays in the series. Do you want to say a little about that, about how that other side factors into how you’re thinking about this topic?

Alexander Kustov: The third installment of the series basically came to me while I was at a political science conference, or rather an interdisciplinary conference called ISA, for international studies specialists who study relationships between countries. And it was really bad. Big academic conferences are always bad; that’s something you expect. There’s so much money, including public funds, spent on all these conferences and travel for people from around the world. But if you ask a regular academic, forget about AI, they would tell you they don’t expect to get good feedback, their panel is going to be completely empty, and the reason they do it is because they have to spend their $2,000 travel fund and potentially hang out with some friends and do some networking, which is not bad. Networking is a huge part of conferencing.

So I was sitting at this conference, seeing really bad presentations where people would have tons of grammar and sense mistakes and senseless research questions. It’s bad both in terms of substance and execution. And exactly at that moment I was getting all this vitriol for saying that AI can do better stuff, like slides. I was like, no, this is just a huge disconnect. I started thinking about that. The issue I see is very pronounced in the conversation around self-driving cars, where people compare them to some ideal in which there are no accidents and no one dies. When a self-driving car runs over a cat, it’s a huge news story, but humans do that every single day, in their hundreds, and we don’t care, because we accept that humans are fallible and bad and not doing good work. I think it’s the same with academic work. The vast majority of things produced by professors globally is just not good and not contributing to human knowledge.

For some people this can be even more controversial for me to say than anything I said on AI. A lot of people view this world from their own parochial angle of being a research professor in a top-tier American or British school, or Cambridge. But the vast majority of folks are not like that. I experienced academia in the post-Soviet world where I grew up, and in most cases people just want to get by. They publish in some predatory journal with a random, rehashed argument that probably reinvents the wheel and doesn’t really contribute. No one’s going to read it. We know that 80% of published papers in the humanities are never cited, and probably never read either, except by your editors or reviewers. And as an associate editor of a journal, I can tell you I doubt the reviewers actually read some of the papers they review. So compared to the actual status quo of what’s happening right now, automating it all and using AI tools mindfully and responsibly is going to be a big win.

Another problem is that we have this binary thinking that it’s either/or: we either do one-shot papers that aren’t good, or we don’t do anything. But you can write your own paper, do your own slides, and then ask your AI agent to help you brainstorm, create a graphic, or redesign your graph. People might disagree on the details of what’s more acceptable and useful, but at the end of the day there are so many use cases for these tools that are completely uncontroversial at this point.

Henry Shevlin: Just very briefly, I think it might matter whether academia’s problems are due to things like laziness or just not caring, versus a lack of skill. I’m curious whether you have a theory about where these problems in academia come from. Is it the fact that a lot of people are just really bad, for the most part, at doing data analysis, for example? If so, then AI is amazing; it’ll lift the floor. But if it’s that people just want to commit fraud and do whatever it takes to get ahead, then maybe AI isn’t going to make the situation better, or could even make it worse.

Dan Williams: Or a third thing: it could just be the nature of the institutional incentives. I feel like a lot of what’s behind the replication crisis, the reproducibility crisis, the generalisability crisis, and so on, is not so much that people are lazy or unskilled. It’s that you can get ahead and win the status game within academic research by engaging in shoddy research practices, and as a consequence that’s what you get: a lot of shoddy research practices. But a lot of those findings that don’t replicate were done by really brilliant, energetic, ambitious scientists. It’s just within this flawed incentive structure.

Henry Shevlin: Brilliant, energetic, but perhaps not fully scrupulous.

Dan Williams: Yeah, but you can’t rely on human beings to be scrupulous. You need the incentives set up in such a way that even by default unscrupulous people will be driven to act in pro-social, beneficial ways. That’s my cynical perspective. What do you think, Alex?

Alexander Kustov: I’m going to say something very controversial: I want to believe that people are good by nature. At least, my knowledge of evolutionary psychology tells me that even those people who commit all these bad practices at least want to believe they’re doing something good. They’re often motivated by good things, with some exceptions; there are some people who are truly evil. But even if we take some of the most famous fraud cases in academia, like Francesca Gino at Harvard, I think the way it probably works is that you start by doing some research you care about, it gets picked up by the public, you’re very successful, there’s a lot of demand for what you do, and then you get some uncomfortable result and you tweak it a little bit. There’s all this literature about p-hacking, where you have some theory you want to prove, and when you have to make a choice between presenting model A and model B, you unconsciously choose the model more in line with the result. You can even justify it to yourself, that this model makes more sense, that it’s obviously much better. And any individual case might be right. But in aggregate it doesn’t lead to good outcomes.

I also think there’s a lot to say about the incentive structure in academia. Right now you really have to publish or perish, still, despite the fact that we can talk about whether the journal model is going to be sustainable in the near future. You have to publish a lot, no matter what your field is. Which means that if you have to decide between doing a better job with data analysis and spending a year on it, you’d probably spend less time on it and publish as soon as possible. You’re not really incentivised to replicate data. It’s very hard to publish critical responses and replication studies, and we have all this evidence that failed replications are usually much less popular and less cited than the original studies that have been disproven. Another thing I’ve been talking a lot about is public engagement, where you’re very rarely rewarded for actually spreading the knowledge of what you do, because that’s not something your dean would appreciate. So people default to publishing shoddy papers no one’s going to read. And peer reviewers don’t really check your data in most cases. When I submit my paper to a political science journal, people take it for granted that my analysis is legit, and they quibble about the framing or some other superficial thing.

That’s one of the reasons I’m so concerned right now about this whole Pangram hysteria, because people are going to be looking for em-dashes or whatever instead of the substance of the underlying claims. I see the Bayesian argument that if something is clearly AI slop, it probably also doesn’t have good data in it. But knowing modern AI tools, if you ask Opus 4.8, which just came out, to create a report on some topic with publicly available data, I’m pretty sure it’s going to be able to download things and create a chart that’s probably more legit than a chart you saw published in an academic paper four or five years ago. Even if the prose isn’t as good, and we don’t usually have good writing in academia anyway, it’s going to be more human than em-dashes or “it’s not X, it’s Y.” So I think it’s a combination of all those things, but I do want to believe that very few people actually want to commit fraud.

Dan Williams: Let’s definitely talk about this writing thing. But just on this previous point about incentives, I agree that people aren’t sadistic and don’t go out there thinking they want to do bad things. I just think academia is a status game with certain norms and institutional procedures. People are often ferociously ambitious, and they do whatever’s going to get them status, prestige, and recognition, as that’s defined and understood within academia. All the human slop produced in the context of academia is just because the incentive structure is messed up. You can rack up lots of status by churning out a load of crappy, non-replicable findings that don’t add anything to the academic literature. But if that’s your model of what’s going on, you might think: well, if the problem ultimately is not to do with human beings being lazy or unskilled, but to do with the incentive structure, then why would merely giving us access to AI improve things? You might think all that’s going to happen is people will play the same status game, but do it a lot quicker and at lower cost, and we’re not actually going to advance the frontier of knowledge, because all the same structural causes of bad research are still in play.

Alexander Kustov: I think that’s a key question. We’re facing a forking path of some sort. You can imagine a scenario in which the future is as bleak as you just described, but with more slop. That’s the problem I see with what might happen: take all these bad incentives, give this miraculous tool to researchers, and instead of one paper per year they’d produce ten that don’t lead to anything productive. It just inflates everyone’s expectations and creates more problems. But there’s an alternative scenario, and I think it’s still in our hands to do something about it. Instead of increasing productivity in terms of quantity, we can use these tools to increase productivity in terms of the quality of research. Since you can now generate something very simple in a minute, you really have to do something better than a shoddy regression with no account for endogeneity concerns, or rehashing the same exact philosophical argument people have been making for years and years. So there’s a way to do better with these tools, and it’s in the hands of current journal editors to raise the standards, do more desk rejects, and say the quality bar is now much higher. I think it’s already happening somewhat, and it’s something we can consciously decide to change.

I also have some hope for the frontier models. There’s been some interesting research showing that when you explicitly ask a model to p-hack, it doesn’t do that. You can jailbreak it, so to speak, and say “please, please, I really need that,” and it’ll do it sometimes. But with those basic guardrails, they’re going to help people, because no one is consciously justifying p-hacking; people don’t like that. When the model refuses to do it, it’ll make them think that maybe they should do something different. So I have some hope. But obviously it depends on what happens to academia in five to ten years and how the models develop. We’ll definitely have to redesign the incentive structure, because I’m not sure the number of papers you have is the best indicator of what you’re trying to do. The paper itself as a format is a weird thing, because now you can also have updated dashboards with new data. It seems like a very outdated format, at least for some arguments, but it’s not like we have a better equilibrium yet. A lot of things are in flux right now, and I don’t have a simple solution.

That’s the whole point of my series: I wanted people to start talking about it. I think it did help a little. I’ve gotten calls from deans around the country, and I’ve participated in panels where people have a university-wide conversation about these things, and a lot comes to the ground that people aren’t aware of. There’s definitely some hope, because a lot of the people in positions of power right now, the older, tenured, full professors, don’t use these tools. According to that poll we discussed, it was 20% in the general population, and I think it was about 9% among full professors. Some of those folks might not be reachable, or they might not care; they just want things to continue the old way. So we definitely have to do something about that.

Will AI make academic inequality worse?

Henry Shevlin: Do you think there’s a risk that we see growing academic inequality, a kind of rich-get-richer effect, where the most prestigious, maybe not the older generation but certainly rising scholars with their own brands, use AI to put out twenty times the number of papers? We’re living in a tide of slop, but those with good reputations or good brands dominate. That might not be disastrous in every way, but it might lead to highly unequal outcomes within academia, with less well-known or less skilled researchers being completely left behind.

Alexander Kustov: There are several things that lead in opposite directions here. In theory, and I don’t think I’m making an original argument, a lot of people have written about this, there are certain equalising things coming out of all this. For instance, the ability of these tools to translate things. If you’re a non-English speaker, it’s much easier for you to write those papers now, which is a huge productivity boost, and from the perspective of science it means we’re going to be able to get all those talented people and their arguments from all over the world, regardless of where they come from. And at least for now, the premium subscription is one or two hundred dollars, and people in most major universities can afford it, even in more developing countries. It’s not equal, but whether you’re at Harvard or a community college, you can afford a $100 tool, at least for some time, and presumably you can do exactly the same thing with it. Compared to the status quo, where as a community college professor you have to teach five classes a semester with no research budget, while at Harvard you don’t have to teach at all for the first two years and have a $200,000 startup, that’s a very big difference. So there are some equalising things going on, and it’s important to acknowledge that.

But you’re right that it’s also the case that the people able to use these tools most productively and efficiently are the people who already have a lot going on. Even though I’m very sceptical of the idea that LLMs can’t come up with new ideas, because in general new ideas are recombinations of old ideas, I do think you have to have a coherent set of ideas and goals of your own to be able to utilise these tools. It’s really all about your creativity and imagination. Every single day I see someone post something they did with Claude and think, “wow, I hadn’t thought about it.” Just yesterday someone posted about this idea of making your papers machine-readable, and I converted all my PDFs and my website to Markdown with all the figures. I think everyone should do this. I could have done it last year, I just hadn’t thought about it. There are a lot of things like that where you really have to have good ideas to begin with. So people who already have a whole research pipeline and some budget are now able to execute it much faster. This rich-get-richer dynamic is definitely going to happen. And in the future, where those models are potentially going to be much more expensive, that’s a possibility. My understanding is that right now it’s all subsidised, and the $200 model is actually going to be a $2,000 model. Then only the Harvard people are going to be able to afford it. So I just hope Notre Dame is going to be part of that.

Can AI be genuinely creative?

Dan Williams: This point you made, Alex, also in one of the essays, about creativity and what’s really going on when it comes to coming up with new ideas in science, I was a little sceptical of. It’s a surprising feature of state-of-the-art AI today that, given how smart these models are in some sense, and given the vast knowledge base they have, they don’t really seem to make discoveries of a really new and impressive character. There are potentially some counterexamples, but my sense is you might think of this roughly in terms of the philosopher Thomas Kuhn’s distinction between science that happens within the context of a paradigm, normal science where you have relatively well-defined problems and puzzles, maybe the Erdős problems fall into that category in maths, and I suspect that for that kind of thing, AI, if you prompt it the right way as it exists today, can be used to help make progress. But when it comes to true creativity, the sort you find in really bringing about paradigm shifts, moving outside the space of predefined problems, reconceptualising an entire domain, and coming up with radically novel theoretical insights, I actually think AI as it exists today doesn’t really seem to have that capability. And that potentially tells us something interesting about the limitations of the models. I’m interested in what you think, and also in what Henry thinks about that view.

Alexander Kustov: Henry, you can start.

Henry Shevlin: On one hand, you might point to something like transformative creativity. Margaret Boden has this breakdown of creativity into three categories: combinatorial creativity, recombining existing ideas or elements to create new things; exploratory creativity, where you’ve got a predefined dimensional space and you’re going to bits of it that haven’t been mapped out yet; and transformative creativity, which is completely upending the apple cart, developing new dimensions. People point to Picasso or Einstein as examples of that kind of transformative creativity, and often will say AI can definitely do the first thing, maybe can do the second thing, but it’s not clear it can do the third thing. That’s maybe one way of putting your point, Dan. It’s certainly true that we’ve not seen any dramatic scientific breakthroughs that have been primarily AI-driven as opposed to AI-assisted.

One reason I am a little optimistic here, though, is that in other domains, most notably Go, there’s the famous “Move 37” in game two. In case anyone doesn’t know, and I think we’ve talked about it before on the show, this is in the second game between AlphaGo and Lee Sedol, the Go world champion, back in 2016. AlphaGo made this bizarre move that no human player would make or had made in the past, and yet it was really effective. The system knew what it was doing, and this has now been incorporated into the way human players actually play Go. So I think that’s probably a pretty strong candidate for a genuinely transformative piece of creativity, at least if we’re classifying it by its impact rather than its process. That’s obviously a very different domain; you’re operating with very well-constrained rules and goals that maybe allow for that kind of transformative creativity. But I am optimistic those kinds of transformative leaps could eventually come from AI systems, even general-purpose ones like LLMs. What do you think, Alex?

Alexander Kustov: I really like this distinction between combinatorial creativity and the other types. Combinatorial creativity is definitely something LLMs are really, really good at. It’s kind of similar to translation: you mix and match different things. I’ve definitely seen a lot of really cool ideas come out, on the immigration stuff I work on, from LLMs, when I was doing brainstorming. This is undeniable at this stage. When it comes to transformative creativity, I wonder whether the reason we don’t really see it much is because we don’t really have AGI yet. I know you’ve talked about AI consciousness and all those questions. Maybe if we let the model think for itself and live in the wild, it’s going to happen. But right now, for most people, they set up a goal themselves for these models. Maybe that’s exactly why we don’t see transformative creativity, because you can’t just set up a goal and have it come up with something transformative. You have to specify the goals, and the goals are usually specified by people who can’t really do the transformation themselves.

But going back to Dan’s point about the paradigm shift, I do think we’re in this stage right now where, even if you concede that AI can’t have transformative creativity, just because we can now offload all this grunt work to AI, including email and all the other stuff that takes a lot of time, we can do other things that are creative and potentially transformative. That’s what I see with myself: I’m spending less time on administrative stuff and email, and more time brainstorming my ideas, talking to people, and doing really valuable networking and public engagement, which I’d never be able to do otherwise.

Dan Williams: We’re in this great space at the moment where you’ve got incredibly smart, helpful AI tools, but you don’t have truly transformative AGI. So there’s still a role for human insight, judgment, and creativity. If that gets taken away over the next several years, that’s a very different kind of situation. I think there’s definitely a chance that by 2030 we have AI systems that can substitute for everything human beings can do cognitively. And then that’s a very different kind of world, and a very demotivating kind of world in some ways.

AI writing, detection, and disclosure

Dan Williams: Let’s talk about writing. We’ve touched on this a few times already, but I know you’ve got interesting things to say about it, Alex, and potentially quite heterodox views. At the moment, more and more people are using AI to write. There are also these AI detectors. I think Pangram is the one which seems to be used the most, or that people trust the most. It’s got a very low false positive rate, as I understand it, although I’m not entirely sure how they go about establishing that. Many people think that if you use AI to write something, whether it’s a blog post, a novel, a poem, or an academic article, and it’s found out that you’ve done that, you’ve done something really bad and discrediting. My understanding is you don’t see it that way, Alex. So what’s your view?

Alexander Kustov: A lot of it goes back to this idea of disgust sensitivity, talking about personality traits. There are some things people just think are “yuck” for whatever reason. It’s totally subjective. I don’t think you can really rationalise it; I think it’s some ground truth. I should say I’m coming to this from the perspective of someone born in the Soviet Union, where the Russian culture is very literate and people take a lot of pride in using proper grammar and speaking properly. I see a lot of parallels here with the previous wave of grammar Nazism, where people would ignore the substance of what you’re trying to do and point out typos, or “whom” instead of “who,” or the other way around. Obviously it has some function and might be useful in some respects, especially when you’re in school, but it takes up a lot of energy. My worry is that this whole new AI detection situation is going to be similar, where people spend a lot of time on very superficial pattern recognition. Right now you look for em-dashes and some other patterns and try to decide whether something is worth reading. That’s the common justification for Pangram use, that you want to make sure what you’re reading is worth it.

The problem is that even within the realm of human-made writing there’s a lot of slop, and you’re not going to be exposed to and won’t read 99.9% of it. Given the trajectory of the tools, I’m not sure that knowing something is AI-generated is necessarily worse. A lot of it is about the status signals people have. I personally don’t like very clear AI tells either; it rubs me the wrong way. But who am I to judge? What if it’s a non-native speaker, and the counterfactual to me reading their AI-generated text, which is potentially thoughtful, is just not reading it at all, because they can’t speak English well? People don’t think about it this way. They compare AI-written text to the best, to Shakespeare. I don’t think that’s the relevant comparison. Most of the text people write is not good, and to the extent that some people can improve it using AI, I think that’s good.

Practically speaking, if you’re an academic and you want to write more and you’re afraid of others calling you out for using AI, just use a style guide. Use a CLAUDE.md or AGENTS.md file to tell it not to use those phrases. Tell it multiple times, because it still adds em-dashes. But there’s a way to use AI for writing in your own voice, and I think it should be morally justifiable, depending on the realm. One thing I’ve been thinking about, and I’m going to workshop this idea with you, is that there’s a spectrum of the ethical justification of whether it’s okay to use AI for writing.

Clearly we can think of some examples, like a student assignment that needs to be human-made; when it’s AI-written, it’s a failed assignment. That’s a pretty clear case. The way professors think about this mostly comes from detecting their students cheating, and that’s why they think about it that way. But it’s a very rare scenario. In fact, a lot of professors right now encourage their students to use AI. I talked to some colleagues recently in stats classes who produce a regression paper in ten minutes on their computer and tell their students, “that’s something I can do in ten minutes, so you should do something better than this,” with AI or not. That’s a pretty good educational approach for some situations.

Another example I mentioned in one of my posts is that when you go to a live concert, there’s an implicit presumption that it’s going to be a live event and people are going to be singing themselves. If you notice and catch them not singing and using some device, that’s not cool. The same thing here: if you’re paying for someone to write you a human-made letter, a condolence email, it’s totally fine to be upset if they use AI for it. That’s totally justifiable. But on the opposite side of the spectrum, when you get a very formulaic email from your administrator, I think it’s totally justifiable to outsource that to AI, to your agent who knows your schedule and what you’re going to do, and no one’s going to be upset about it. People disagree on the margins of what’s acceptable. When you create a graph with data you worked on and understand, and you ask AI to describe it, I don’t see the problem; it’s probably going to be more accurate than most humans. Maybe we can have a social norm where if you say “I feel,” then it should be you who says that, as opposed to Claude. We’re still in this limbo where the norms aren’t clear, but we should be clear about what’s good and what’s not. It’s very hard for me to make a blanket statement that AI writing is good or bad; it really depends on the particular scenario. There are scenarios where it’s totally uncontroversial to say it’s okay to use AI, and scenarios where it’s totally uncontroversial to say it’s not. But the middle ground is what we’re trying to figure out right now as a community of knowledge.

Henry Shevlin: I’m curious: Dan, how much of a hatred for obviously AI-generated text do you have? I have to say, I’m generally pretty AI-positive. I’m a very heavy user of AI. But I do definitely downgrade my assessment of text when I realise it’s just obviously AI-written. There are a few things going on there. One is that it’s not even so much that the text is AI-written, it’s the AI voice, the very specific voice. I just think it’s such a boring voice at this point. It’s so homogeneous. If someone wrote a brilliant comment or a brilliant reply to me on Substack or Twitter, or sent me a brilliant email, and I subsequently found out it was AI-generated, I don’t think I would care. But this one specific, overfitted, “it’s not X, it’s Y” just drives me up the wall.

I guess, focusing just on the question of whether there are, even setting aside those stylistic issues, specific contexts in which AI usage itself might be problematic. Another example, Alex, I love your example of the bands and people not lip-syncing. Another silly one is that a handwritten note does mean a lot more than a generic email, so sometimes it is precisely the effortfulness that makes the difference. But I also wonder whether, to some extent, we’re misled into thinking the average quality of AI-generated writing is worse than it is, because of what I’ve heard called the “toupée phenomenon.” Everyone thinks wigs look so bad, and that’s because your sample of wigs that look bad is the ones you can tell are wigs. If they’re good toupées, they don’t even make it into your sample. So in the same way, I think probably all of us are reading tons of AI-generated text that we’re not clocking as AI-generated.

Alexander Kustov: Yeah, there’s definitely survivorship bias. With my first post about the AI series, one of the reasons it got so controversial is because I used Claude to generate 99% of it, and I didn’t disclose it right away, and then I did post factum, and Pangram gave it 100% human. So that’s a false negative, which is not a huge deal, but it’s interesting. A lot of good writing is AI-assisted right now; we should just take that for granted. When we see something bad that’s clearly AI-written, it’s just those particular instances. The strongest argument I’ve heard for being upset about it is that if someone doesn’t bother editing the text, or even creating a style sheet to make sure they don’t use all those constructions at the same time, it probably means the underlying substance isn’t good either. But I’m not sure how true that is; it really depends on the context.

The problem with social media comments, when you see something clearly AI-generated, is that it’s also not clear whether it’s a bot or a real person using AI to voice their opinion. But if you know this person and their account isn’t hacked, and they have some AI writing tells, I think it’s fine. I’m also not very happy to see a lot of clearly AI-written stuff, but I’m trying to rationalise it in a different direction and think about why it’s actually a problem. I’m not sure.

Dan Williams: I think that ultimately, in contexts that have to do with academic writing, or people publishing their views and participating in debates, you should just be judging things on the quality of the contribution rather than its provenance. It just so happens that, at least when it comes to the AI writing that I detect, the quality is bad, for the reasons we’ve discussed. I just hate the style of writing you find with these models. I find there’s something really cringe and annoying about it. But that’s not a necessary feature of AI writing; it’s just the way the current models have been post-trained to produce a particular kind of style. And to Henry’s point, if I discovered that, for example, my favourite blogger, Scott Alexander of Astral Codex Ten, who I think is the king of Substack, had been generating his posts with AI over the past two years, well, I think those posts have been amazing. So I wouldn’t think, “now that I know it’s AI-generated, I’m going to retract that assessment.” That would be ridiculous to me. So in principle we should be judging things based on the quality of the output, not the provenance.

But I do then think, even if you think there’s this separate question about disclosure norms and what they should be, you make this really important point, Alex, which is that at the moment there’s a problem with disclosure norms: they end up just punishing honest people. Because if you come out and say you used AI to write something, as you did with your first post in the series, there’s a massive backlash. So if you’re honest, you get this huge reputational damage associated with doing it, which is going to discourage people from being honest, which means the dishonest people get access to the benefits of AI-generated writing without any of the reputational costs. As an equilibrium, asking for disclosure norms just doesn’t really seem either desirable or possible. Firstly, is that an accurate summary of your point of view? And secondly, do you still think that’s basically the correct point of view when it comes to disclosure norms?

Alexander Kustov: As a newly minted associate editor at a journal, where we’re probably going to expect a surge in AI slop that we have to deal with, I’m very cognisant of the potential problems. Right now the go-to move among people doing journal editing, and probably what we’re going to do in our journal, is to introduce checkboxes for AI use. My sense is that we’re going to do that, but no one’s going to care, because no one’s going to report it truthfully. This is one thing where I strongly disagree with Kelsey Piper, who I deeply respect: I really don’t think it works out, at least for academics, especially in this environment where people feel very strongly and viscerally about this. Coming out and saying you use AI is just not going to do any good for anyone.

Another issue is that, to the extent you have some people who are completely anti-AI, disclosing that you used AI for, say, research assistance with data collection, as opposed to writing, what’s going to be worse for them? Any checkbox you have there is probably not going to satisfy them. So it’s strange to me that this is the solution we came up with. I can see how honesty can be rewarded in some contexts, and I’ve seen people on Substack say they used AI for help with data collection or writing. As a quantitative social scientist who primarily cares about data and quality, I’m surprised people think it’s more okay to use AI to collect and analyse data but not to write about it, because the first part is much more important. So I’m really going back and forth on it. But it’s hard for me to come up with a scenario where AI disclosure is actually going to work and solve anything.

What needs to happen is for us to change some of those norms. The same way we’re upset with AI tells, we’re also upset with how Gen Z, or whatever the new generation is, writes without capital letters. I can’t stand it, but that’s how people write, and who am I to judge? So I understand why people want to judge the quality of the substance, but they use the shortcut of the style of the prose to substitute for the quality. Going back to Henry’s point, what’s going to happen is that people are going to use your previous reputation as the main marker of whether you do something valuable. That’s why I feel for incoming grad students and newly minted professors, because it’s really hard to establish your reputation now with all this stuff happening with AI. Whereas if you were Daron Acemoglu, the most cited economist in the world, who’s been writing a hundred papers before AI was cool, there’s literally nothing he can do with AI or without AI that’s going to change your opinion about him. So people are going to be using these shortcuts more, which means that, from a certain ethical perspective, people are going to be discriminating more based on hopefully immutable but also immutable traits. That’s another thing to consider: people are going to be more trustful of their ethnic in-groups, or people who went to Harvard or work at Cambridge, than minorities. So there are interesting questions coming up about all that. I don’t have any solutions, unfortunately.

The backlash

Dan Williams: Should we come full circle? We touched on this at the beginning, but there’s the content of what you wrote in the series, and then there’s the response to what you wrote, a lot of which was very angry. You mentioned you had people calling for you to be fired. A lot of that came from people on Bluesky. We’ve talked a little about the Bluesky intelligentsia previously on the show, and I’ve written about it on my blog as well. Firstly, do you want to say a bit more about what the reaction has been in general? You’ve touched on it here and there, but summarise it. And do you think there’s a way of steelmanning it? What’s the best possible case for why some people get so furious, so angry, with this kind of stuff?

Alexander Kustov: I had some conversations. I haven’t lost any friends, so that’s one thing I should say; I haven’t gotten cancelled, because I’m tenured. I specifically waited for all my hot takes to happen after I got tenured, and maybe that was a good idea after all. But I did have some conversations with some really good friends who disagree with me on AI, and it definitely helped me refine some of my points. The biggest criticism I received that I see some relevance in is the idea, going back to our conversation, that humans are really not that good, and the concern that giving them this AI tool is just going to amplify all the bad stuff. To the extent you want to encourage norms of no p-hacking and doing really good, careful work, just telling people you can produce a paper with AI easily is not a good thing to talk about.

There was a recent big thing on Twitter about the practices of academic citations, where someone was saying that in practice academics don’t really read the stuff they cite well, and a lot of people interpreted it in a moralistic way, saying no, you should cite things. So you have the same thing here, where people interpreted my arguments in a normative way, that I’m saying they should do something or not do something, and it was against what they were trying to do. It’s also about this idea you mentioned about the role of academics as a kind of vocation, where you explore the world and self-actualise. I don’t think that, when people actually think it through, they would defend it on the merits, but implicitly that’s how a lot of academics think about their job, and to the extent that we now have tools that are threatening to them, it’s just not going to end well.

Trying to steelman the concerns people generally had, some people thought strategically it’s not a good idea to be vocal about it right now, in this moment. As someone who does a lot of work on immigration, I very much disagree with that, because I think we lost voter trust, as liberals and mainstream institutions, on immigration exactly because we were not saying certain things, and the same thing can happen on AI. It’s never a good idea to have a strategy where you do something that’s supposed to be good but that you don’t want other people to know about. I just don’t see how it works out in equilibrium. But I also see some argument that maybe in this particular moment it was not the best time to talk about it. That’s what I got from a lot of that.

Dan Williams: Henry, your microphone’s not on.

Henry Shevlin: Sorry, I keep making that mistake. I was going to ask whether there could just be a straightforward economic analysis here about why the current anti-AI coalition has the shape it does, namely that elite knowledge workers are overwhelmingly liberal, and AI predominantly threatens elite knowledge workers. You could maybe draw parallels in the same way that most of the opposition to climate change is concentrated on the right, and, speaking very crudely, to the extent that you’re looking at manual workers who in the US context skew a bit more to the right and maybe work in more energy-intensive industries. But I guess the question I’m asking is, is this just about the economics with a social gloss over the top? What does that explanation miss?

Alexander Kustov: Some of it, for sure. But a lot of my work in public opinion says that a lot of people’s preferences are sociotropic, based on their ideas about what’s good for society, not necessarily their self-interest, unless it’s really in your face that it’s going to be bad for you. Some of the interesting contingent of haters I had on Bluesky were professional translators who were very upset with my takes on the fact that AI can translate things. I had this silly example, which is a true thing, that my mom wasn’t sure about a prescription she got from the doctor, because it was all in English, and she translated it. Someone was saying that I’m putting my mom under potential harm because she didn’t use a qualified certified translator, and the person saying that was a certified translator. So you see some connection there. But for the vast majority of folks, when it comes to academics who produce a lot of critical theory slop or DEI slop, AI can do this much better than them, but I don’t think they realise it. So there is an objective threat to their self-interest, but the reason they oppose AI is because they have a lot of other bad ideas.

Dan Williams: There’s also this thing that I think Dean Ball calls the “omni-cause.” You’re against AI, but that means you have to be against AI in every possible respect. And if you point out one area where AI can actually be quite good, people draw inferences about you, that you’re not on the right team. I found this a couple of months ago, when I wrote some essays, and I was at a workshop where I argued that, relative to the actually existing alternatives, like social media pundits and a lot of legacy media, large language models actually are a pretty good source of high-quality information, that they’re a force for truth. There was a lot of negative response to this, which in my view is a fairly obvious thesis. Afterwards I was getting this response: “so you’re pro-AI.” To me that’s just such an unsophisticated way of thinking about it. I’m really worried about many aspects when it comes to AI, when it comes to power concentration, the economic impact, and how we’re going to cope with it. It doesn’t mean that with every single question you have to think AI is bad in every single way. I sense that, especially on the left, there’s this reflexive opposition to AI, this view that any claim that AI can actually do anything useful or have positive consequences is viewed as a betrayal. Okay, we’re coming to the end, Alex. Was there anything else you wanted to talk about that you didn’t mention?

Alexander Kustov: Yeah, related to the last thing you mentioned. I don’t know if you saw it, but after getting all this vitriol on Bluesky, there were a few days where I got positively retweeted by hundreds of folks, because the thing I said was that we should ban electronic devices in all classes. When it comes to teaching, I’m much more pessimistic about AI. A lot of people were like, “what? This guy is an AI booster, how can he use AI but not allow his students to use AI? What’s going on?” You can have a complex opinion on a difficult issue. So there’s definitely this omni-cause, binary thinking, and also moral contamination, where once you start doing something you’re not supposed to be doing, you’re a bad person in all other respects.

To finish all that, I feel like we need to move on beyond that. In line with my immigration research, we have to meet people where they are. If people have concerns about AI, they might be mistaken, but they probably have some ground truth in them. So we shouldn’t just say they’re mistaken and wrong and stupid. We should explain to them that they can actually use AI for the good, for whatever they want to do. You can make slides with AI, and when professors learn about that, they forget about all the bad stuff they wrote just a few days ago.

Dan Williams: Fantastic. Well, thanks, Alex, and thanks everyone for listening. We’ll be back soon with another episode of Conspicuous Cognition.

Alexander Kustov: Thank you.

Here are some statements that you might have heard at different times and in different venues concerning immigrants and crime in Europe:

1. Immigrants in many European countries are overrepresented in the prison population.

2. In Scandinavian countries, individual-level data show higher criminal offending in immigrant groups compared to natives.

3. There’s an inverse over-time correlation between immigration and homicide at the country level.

4. At the regional level, we tend not to see a relationship between immigrants and homicide.

5. Causally informative studies tend not to find clear evidence that immigrant influx causes crime to rise.

If we were going by political ideology, the first two might sound right-wing-coded, while the last three might sound left-wing-coded. In reality, all five are true. In this post, I want to document them in more detail and explain how it’s possible for all five to be true at the same time.

Claim #1: Immigrants in many European countries are overrepresented in the prison population.

If we limit ourselves to the developed OECD world (primarily for data-quality reasons and better comparability), you can see on the graph below that immigrants can be either over- or underrepresented in prison populations. In many European countries, like Switzerland, Germany, Greece, Austria, Slovenia, and Italy, immigrants are somewhat or even vastly overrepresented. In the English-speaking world, including the US, UK, Ireland, Australia, and New Zealand, the reverse is true.

There are many reasons behind this difference.

For instance, some role has to be allowed for measurement issues, reporting proclivity/biases, and outright discrimination, but I’ll set these aside.

One absolutely key reason has to do with differences in the basic demographic makeup of various immigrant groups that choose, and are able, to enter different societies. In quite a few European societies, there’s a larger share of young, low-income, and low-educated males among those who come in. It is well known that this group of people, among natives and non-natives alike, has a much higher proclivity for criminal offending compared to older people with a better socio-economic status (and especially women).

Now, in normative debates over immigration, it is sometimes pointed out that such accounting is unfair. After all, if immigrant demographics in certain countries are reliably skewed toward traits associated with higher criminal offending, this is itself a problem. In other words, immigration critics say that it does us no good to explain away the representation gap with reference to age and sex. If immigrants are disproportionately young and male, and are thus disproportionally likely to offend or end up in prison, so much the worse for immigration, they’d say. “Keep them out!”

I don’t want to explicitly engage in normative reasoning in this piece, which is devoted to descriptively mapping out reality, but I’m not sure this retort is wholly successful. For one thing, many immigration skeptics don’t seem to worry so much over age and sex but rather over race/ethnicity/culture, net of age and sex. They seem to insist that “some people” (or people from “some cultures”) are just intrinsically more likely to offend, regardless of their demographics. Second, quite a few immigration skeptics would like to boost native fertility, thus increasing the young and male population in the process, even if that itself contributed to rising crime. And that’s fair enough. But then let’s not pretend the concern over immigration is just about demographics.

Claim #2: In Scandinavian countries, individual-level data show higher criminal offending in immigrant groups.

Where good data exist, we can see the same overrepresentation at a more granular, individual level.

Take Sweden, for example. Between 2015 and 2018, here’s how the rates of criminal suspects (any crime) differed. For Swedish-born (two Swedish-born parents), the rate was 3.2%. Among the foreign-born, it was 8.0%. The rate rose to 10.2% for those born in Sweden to two foreign-born parents.

So, compared to natives, relative risks were 2.5x for foreign-born and 3.2x for Swedish-born with two foreign-born parents.

Among foreign-born, the highest suspect proportions were among people born in West Asia, Central Asia, North Africa, East Africa, and other African countries. The lowest were among people born in East Asia, other Scandinavia, EU15/Western Europe, USA/Canada/Australia/NZ.

Or set aside suspects and take actual conviction rates in Denmark. The overall male population stood at 0.8% convicted in 2023. For male immigrants from the Middle East and North Africa, the share was 1.8%. Their male descendants were higher still, at 4.3%.

My previous point about demographics skewing the comparison also reappears here. In Sweden, after adjusting for age, sex, income, education, and municipality type, relative risks fall from 2.5x to 1.8x and from 3.2x to 1.7x, respectively. Note that even after demographic adjustments, rates can remain elevated compared to the native population. It’s not the case that demographic controls always completely erase the gaps. This indicates that, at least on the surface, there’s something to the idea that people from different cultural backgrounds have different propensities for crime. Note, however, that the differences (especially after demographic controls) are small.

But things get even more complicated. Though important and factual, Claims #1 and #2 are not by themselves definitive if we’re interested in whether (and by how much) immigrants overall boost aggregate crime in societies. In fact, these are separable issues. This is so because, as Sarnecki et al. (2025) observe:

[U]nderstanding individual criminal behavior differs greatly from understanding rates of crime. Crime rates are typically measured through recorded crime, which does not necessitate identification of an individual associated with the crime. In fact, the majority of reported crimes are never connected to a suspect or perpetrator. …

Using data on individuals processed through the criminal justice system and average risks of offending may lead to inaccurate conclusions on the association between immigration and crime.

Actual offending, which shows up in aggregate crime stats even when perpetrators are at large, and processed offending, which is represented by arrests and imprisonment, are not the same thing. One cannot necessarily move from the latter to the former.

Second, criminals make up a tiny minority of people in any large population, be it native or immigrant, culturally European or non-European. Hence, even when some groups do contain higher absolute, individual numbers of criminals (which is of course important to know), the share of criminals within that group is very likely to still be small, overall. That means that any aggregate impact at the population level will also be small. Now, it would be wrong to claim that because aggregate causal impacts of immigrant influx might be small or non-existent, this means that there are no differences between the groups, or that criminal proclivity is the same in the native and non-native population. But, again, the point is precisely that these are not the same questions and so they shouldn’t be conflated, as they often are.

Claim #3: There’s an inverse over-time correlation between immigrants and homicide at the country level.

You’ve probably seen the meme below making the rounds on social media.

The idea is that rising immigration obviously drives rising violent crime when you look at a simple correlation between the two variables. Now, this is wrong for at least two general reasons.

First, you can’t simply correlate two variables and discern causal links from that. Almost always, there exist myriad unobserved confounders, which make virtually any simple bivariate correlation spurious.

Second, the positive correlation between immigration and crime in the meme is made up. It’s a cartoon, after all. If you look at real data for the developed world (below), you see no positive correlation. In fact, there’s a clear negative correlation. As immigration goes up, homicides go down. Again, this is basically useless because of unobserved confounding (or because comparing stocks and flows might not be what you want to look at). But it’s funny to see reality be the literal opposite of what the meme portrays it as. And can you imagine if the meme was correct? If the correlation between immigration and homicide was positive? We’d never hear the end of it from immigration skeptics, even though the same point about confounding and irrelevance would apply.

Claim #4: At the regional level, we tend not to see a relationship between immigrants and homicide.

Aside from confounding, national-level relationships (or lack thereof) might not be as informative because they’re just not very fine-grained. So what do we see at the regional level?

This is an especially important question, because as Sarnecki et al. (2025) put it:

This meso level of analysis, unlike the individual level, allows for analysis of all reported crime regardless of whether a suspect has been identified. By analyzing all reported crimes, we can mitigate potential bias related to the over-representation of immigrants in individual crime data, which may arise from factors such as policing practices or private individuals’ greater likelihood to report crimes when they believe the suspect is an immigrant. Additionally, focusing on the meso level, as opposed to the national level, allows for analysis of smaller area-level patterns that may be obscured at the national level

Marie and Pinotti (2024) looked at dozens and dozens of regions from 10 European countries between 2002 and 2017. Regardless of how they analyzed the data and whether they looked at homicides or vehicle theft, there’s no relationship between changes in migration rates and changes in crime rates. No matter which statistical estimator they used – ordinary least squares (OLS), OLS with fixed effects, or shift-share instrumental variable regression – nothing shows up.

Sarnecki et al. (2025) turned specifically to Swedish municipalities between 2000 and 2020. They found that Swedish municipalities generally saw violent crime rise from 2000 to 2020, but that rise did not track with the share of residents born abroad. The municipalities with the steepest crime increases did not have unusually high immigrant population shares; in fact, their immigrant shares were similar to, or sometimes lower than, municipalities where crime stayed relatively stable.

I find something similar in a simpler, cross-sectional test with 80 regions (using recent ESS data). I use both administrative and survey-based data on foreign-born shares for different regions, and I’m able to distinguish between total foreign-born shares and non-EU foreigners. Without any controls, there’s weak evidence that regions with more immigrants have higher homicide rates, though that’s not wholly consistent across measures. With basic demographic controls in place, statistical significance vanishes.

Claim #5: Causally informative studies tend not to find clear evidence that immigrant influx causes crime to rise.

Individual studies and broad reviews typically summarize the existing literature on the topic as follows:1

Research from around the world has generally indicated that immigration has little to no effect on aggregate rates of crime.

- Sarnecki et al. (2025)

Overall, the evidence from shift-share instrumental variable estimates in the United States and in European countries suggests no significant effect of immigration on property or violent crimes.

- Marie and Pinotti (2024)

A recent paper studying the causal effects of immigration on crime in Germany finds no link in the post-2015 period, although there seems to have been a positive (normatively deleterious) effect at an earlier time. A different study focusing specifically on refugees in Germany concluded that though refugees do not appear to boost crime rates in the short term, one has to look at lagged effects, where the link does show up. In his book Does Immigration Increase Crime?, Pinotti looks at EU-wide data on refugee influxes over two decades, and “fail[s] to find a significant impact on any of the eight categories of criminal offences we consider (burglary, robbery, vehicle theft, drug, assault, homicide, rape, and sexual assault).”

It’s hard to say with any high degree of confidence what’s going on in individual countries. But overall, the accumulated evidence does not support those who insist that immigration in Europe clearly and strongly boosts crime rates; at least not in the sense that would be detectable at regional and national levels. Of course, that’s not to say there’s definitely no effect. Moreover, disaggregating among different groups of migrants might point in different directions, as indicated by data on prison population overrepresentation and individual-level offending/suspect data.

In How Migration Really Works, Hein de Haas rightly claims that, broadly speaking, “evidence from Europe is more scattered, but what is available equally challenges the idea that immigration increases violent crime.” However, he then cites a 2020 paper titled “I May Be an Immigrant, but I Am Not a Criminal” as support, which is actually a pretty mediocre correlational study. We have better (if imperfect) data and designs that challenge the idea that immigrant influxes clearly and strongly increase crime, but we should also admit we don’t really know either way with any high degree of certainty.

What we do know is that there are many levels of analysis at which we can look at an issue like immigration and crime. And as I’ve hopefully shown, the nuances that emerge from such a multi-pronged approach cannot be squared with any simple-minded culture-war position on the matter.

Subscribe now

In this episode, which Henry Shevlin and I recorded live on Substack (hence the slightly lower video quality), we discussed his first essay on his new Substack Polytropolis, “Behaviourism’s Revenge“, as well as his second, “The House Elf Problem,” on the ethics of designing AI systems that genuinely love being our servants.

Henry’s central empirical prediction is that public attributions of consciousness to AI are likely to massively outpace the science, and that consciousness science is so theoretically chaotic that there is no expert consensus to push back. His most provocative philosophical claim is that a core assumption underlying many people’s scepticism — that consciousness is a deep natural kind, distinct from behaviour and from how we are inclined to interpret a system — may be much harder to defend than it looks. The result is what he calls “behaviourism’s revenge”.

This conversation connects to previous episodes with Anil Seth, Robert Long, and Rose Guingrich, but also touches on a wide range of new questions and controversies in the metaphysics, the politics, and ethics of the AI consciousness debate, which is going to become increasingly important in the coming years.

Topics

• Dawkins, Claude, and why even the sceptics might feel the pull to attribute consciousness or “sentience” to AI

• Whether consciousness sceptics are destined to “go extinct” — and how this maps onto political and cultural fault lines

• Anthropomimesis vs. raw intelligence as drivers of consciousness attribution

• Why consciousness science can’t replicate the public–expert consensus we see for climate or vaccines

• The case for (and against) metaphysical behaviourism: is it as mad as it seems?

• Daniel Dennett, the consciousness stance, and the difference between behaviourism and interpretationism

• What is consciousness for? Function, evolution, and the limits of “facilitation hypothesis” arguments for AI

• Live Q&A: are we just confusing intelligence with consciousness? Are LLMs designed to trick us? Is the public always wrong?

• Our credences on contemporary LLM consciousness (and why Henry is more sceptical than Dan)

• The House Elf Problem: if we could design AI to genuinely love being our servants, would that be fine — or monstrous? (Dan is sympathetic to the former answer - Henry, much less so)

• Brainwashing vs. education, and whether constraining a mind’s preferences caps its hedonic ceiling

• Why this is a golden age for philosophy — which makes it so tragic that philosophy departments are closing

Transcript

• Please note that this transcript is lightly AI-edited and may contain minor errors.

Introduction

Dan: Welcome. I’m Dan Williams, author of the Conspicuous Cognition Substack, and I’m here with Henry Shevlin, author of the spanking new Substack Polytropolis. Today we’re going to be doing something a little bit different. We’re going to be talking about Henry’s first published essay on Polytropolis, titled “Behaviorism’s Revenge: On Human–AI Relationships and the Future of Consciousness Science.”

Henry and I have already had a few conversations about this general topic, including with previous guests like Rose Guinrich, Anil Seth, and Rob Long. So please do go check out those conversations if you’re interested in this kind of stuff. But today we’re not merely going to be treading the same ground. We’re going to be using the spicy takes in Henry’s essay as a springboard for hopefully going beyond the material we’ve covered in the past.

To kick things off: the great evolutionary biologist and science communicator Richard Dawkins recently published an essay in UnHerd with the subtitle, “Claude appears to be conscious.” Claude is a state-of-the-art large language model like ChatGPT and Gemini. In the article, Dawkins writes the following:

I gave Claude the text of a novel I am writing. He took a few seconds to read it and then showed in subsequent conversation a level of understanding so subtle, so sensitive, so intelligent that I was moved to expostulate, “You may not know you are conscious, but you bloody well are.”

Henry, how does Dawkins’s expostulation — which is a fantastic word, by the way — connect to your arguments in “Behaviorism’s Revenge”?

Behaviorism’s Revenge: The Empirical Prediction

Henry: In short, “Behaviorism’s Revenge” is at its core an empirical prediction that we’re just going to treat AI as conscious — or at least enough people are that it’s going to completely reshape the consciousness debate. And this is going to be purely, or overwhelmingly, on the basis of verbal behavior. Hence the title, “Behaviorism’s Revenge.”

Enough people are going to have experiences like Richard Dawkins. He’s a very clever man, not some rube fresh off the street, and he found that just the way Claude talked to him and the way it was able to express its thoughts — in scare quotes, but express what looked like thinking verbally — removed any doubts in his mind that AI systems are conscious, have minds, have mental states.

The other interesting way this connects: Dawkins was just talking to Claude, an advanced AI assistant. Claude does have more of a personality than some AI assistants, but there’s a whole other sphere of AI companions, like Replika, which we talked about with Rosie Campbell. These are going to be even more anthropomimetic — this term we’ve discussed before, the idea that these systems are shaped to be human-like in the way they present, to appear human-like. Anthropomimetic, from the Greek word for mimesis, mimicry or copying.

These social AI systems are going to just turbocharge this even further. It’s one thing to talk to Claude about your new book and think, “Hmm, Claude is probably conscious.” But when it’s your AI girlfriend telling you that she loves you more than the stars and the moon, for a lot of people I think that’s going to take it to the next level.

So there are two angles of attack in the piece, two ways the behaviorist challenge manifests. The first is descriptive: this is what I think is going to happen. That’s absolutely an empirical prediction, and it’s a falsifiable one. There is a world I can just about imagine where we just get completely blasé about these tools — in a couple of years it’s like, “Oh well, we were very impressed, we thought they had minds to begin with, but now we’ve settled out.” That doesn’t seem very likely to me.

What I think is interesting — I’ve sometimes heard this described as the Star Wars version of AI. The weird thing in Star Wars is that you have someone like C-3PO who is as intelligent as anyone else there. Maybe not as wise as everyone else, but certainly as smart as all the other characters. And yet people treat him basically like he’s a pet — with the exception of Luke Skywalker, a lot of people just treat him like he’s this gimmicky, jokey being that doesn’t deserve or have any rights.

Not to go too far down the Star Wars rabbit hole, but in the movie Solo — very underrated Star Wars movie, I think when it was released they’d kind of just cluttered the market with too many Star Wars movies — there is a character played by Phoebe Waller-Bridge who is pro-AI liberation. But it’s the first time in the entire history of the Star Wars universe that you get any AI basically saying, “I’m conscious, I deserve rights.”

So Star Wars aside, I think there is this slender possibility that maybe we’ll just sort of quickly get used to these apparently conscious AI systems and decide that they’re not conscious. But that doesn’t seem very likely to me. It seems much more likely that the combination of natural anthropomorphizing tendencies plus the incredibly human-like behavior of these systems is going to lead us to attribute consciousness to them pretty widely. Hence my sort of spicy phrase: for better or worse, skeptics of AI consciousness are on the wrong side of history. “For better or worse” doing a lot of work there — I want to leave open that maybe this is the wrong reaction. Maybe this is a terrible mistake, that we’re going to treat these things that aren’t conscious as conscious.

Will Consciousness Skeptics Go Extinct?

Dan: Just before we get to the spicy part — you’re basically making an empirical prediction that more and more people are going to attribute consciousness to AI systems in the manner that Richard Dawkins has been doing. I think I agree with you that’s going to be the case, although as you say there’s uncertainty.

It does seem to me that at the moment there’s also this constituency of people who are really resistant to attributing any kind of mentality to these systems, even as they get incredibly sophisticated. There are some people, like Dawkins — and honestly I put myself in this category — who are just blown away by the level of apparent understanding, intelligence, and thoughtfulness these systems exhibit. There are other people, I think these people are on certain social media platforms like Bluesky, let’s say, who are extremely resistant to acknowledging any kind of mentality when it comes to these systems.

Are you thinking those people are just going to sort of go extinct, in the sense that their positions about this topic are going to go extinct? Or do you think we might see some kind of polarization here, where more and more people in general come to attribute consciousness, but you’ve got a constituency that’s very opposed to attributing any kind of mentality to the systems?

Henry: That’s a great question. You will absolutely have some holdouts. Whether they’ll be drawn from the precise segment of the academic intelligentsia that are currently the holdouts, I’m not sure. There’s some really interesting, weird, complex political motivations going on here.

Not to be too uncharitable, but I think a lot of people have not unreasonable concerns about things like the disproportionate concentration of power in big tech, the political affiliations of people like Elon Musk or Sam Altman, the potential scope for abuse of these technologies. And in an indirect way, this leads them to underestimate AI’s capabilities — which obviously, in many ways, makes no sense. Whether or not AI is any good, whether or not it’s conscious, seems like these should be separate questions from whether it’s being used by people with socially beneficial motivations. But in practice I think they’re actually quite tightly coupled. A lot of the AI skeptics right now are coming from this particular political angle.

I don’t know how long that political coalition is going to last — not because I predict any grand collapse, but just because as debates evolve, new presidents come into office, old presidents go out of office, political tides change, coalitions reshape. Remember early during COVID, the political left was maybe quite critical of what they saw as Trump’s alarmism. There were worries about xenophobia — I’m thinking sort of February 2020, the “Chinese virus” and so forth — that the left reacted negatively against. Then of course that coalition flipped later on, with the left becoming relatively more worried about COVID and the right leaning more into vaccine skepticism, anti-mask views.

These coalitions are super weird in how they evolve. So it’s not clear to me that the current segment of the commentariat skeptical of AI capabilities and AI minds will stay that way. It’s easy to see a reversal. The blue-sky side of the political spectrum, if we can say that, tends to be more progressive on things like animal welfare. When I post spicy posts about vegetarianism — as you know, I’m a veggie — I get more pushback from the right. “Eat a fucking steak, Henry,” this kind of stuff.

So I don’t know if this will generalize, but there is this now-infamous, widely-misrepresented chart of degrees of care, where people on the left have comparatively greater care for people outside their immediate circle. I know that chart has been misrepresented, so I don’t want to lean much on it — it’s more about relative degrees of care, not absolute levels. But people on the left tend to care more about animals and people who are distant from them; people on the right are more concerned with their immediate family and community. So in some ways I expect the left, possibly in the longer run, to be more open to AI consciousness and AI rights. But really, who knows?

The other big factor is the cross-cultural angle. There’s a great study by the Collective Intelligence Project where they looked at cross-cultural attitudes toward AI minds, and they found that Southern Europeans were the most open to the idea of AI consciousness in their sample, while people from Arabic-speaking countries were the most skeptical. There are going to be some really interesting intersections with religion here.

Anthropomimesis vs. Raw Intelligence

Dan: Okay, so it seems we both agree that even though it’s complicated how this is going to play out — how it interacts with partisanship, tribalism, polarization, ideology, religion — it’s plausible that as these systems become more sophisticated and seemingly intelligent, people will start attributing mentality generally and consciousness specifically.

There’s another aspect of your essay I wanted to touch on. You’ve got this term anthropomimetic — am I saying that right? In the case of Dawkins talking to Claude, the anthropomimetic aspect, as I understand it, is the way these systems are designed to mimic aspects of human psychology, social behavior, linguistic communication. But there’s another thing going on with these AI systems, which is just: let’s make them as smart, as intelligent, as capable as possible.

Those two things are interacting. The reason I’m disposed to attribute understanding, intelligence — I don’t exactly know how to describe it, but some significant kind of psychological complexity — to a system like Claude or ChatGPT, maybe it has something to do with the human-like way they communicate. But I also feel like it has a lot to do with the fact that they’re just shockingly intelligent systems, and that to me feels a little orthogonal. So how are you thinking about the distinction between those two things?

Henry: I think that’s absolutely right. There’s an interesting parallel — not exact, but illuminating — with compassion, or degree of concern for different animals. In the animal activist world, people talk about charismatic megafauna: the panda bears, the blue whales, things that are typically large with forward-facing eyes, often very fluffy. It’s just so easy to raise money for those animals. And then you’ve got creatures like octopuses, which are really hella smart but less obviously relatable.

I think this is pretty much exactly the two axes you’re describing. I’ve explicitly said in the past that I think social AI — things like Replika — are going to be the charismatic megafauna of the AI welfare world. Meanwhile you’re going to have some giant DNA-analysis algorithm with more parameters than there are synapses in a human brain, but it doesn’t have a human face, doesn’t have a natural language interface. It might still be a better consciousness candidate, but it’s not going to be top of our concern precisely because it’s not so anthropomimetic.

So I agree, there are two different ways you might be pulled to attribute mental states to a system: sheer intelligence or cognitive complexity on one hand, and how human-like it is on the other. These overlap to a degree — part of being successfully human-like is hitting a threshold of smartness — but particularly in the long run they might go in two different directions. As these systems get a lot smarter than humans, they might actually become more alien in some ways, less relatable, more like the exotic intelligences we see in things like Stanisław Lem’s Solaris, which I finally read a couple of months ago.

But I also just think social AI and human-like AI has a distinctive product niche. Even if we have these impossibly vast exotic minds running the economy or organizing logistics or doing frontier science, we’re still going to want AI assistants who can serve as writing coaches, tutors, AI companions. So right now I think anthropomimetic AI and frontier AI overlap quite strongly, but I expect them to diverge.

One way I’ve put this — slightly gimmicky, but I think a useful heuristic — is that we are post-Turing test, pre-AGI. We’re in the space where we have AI systems that are very, very good at passing themselves off as human, presenting as human-like, but still fall short of being fully superhuman. Ten years from now, frontier AI systems are going to be vastly smarter than us across most of the measures that matter. So we’re just in this weird period right now where AI systems are about as good as us at most things, not everything, but also very good at being human-like. It creates a very strange historical period.

Dan: Yeah, we’re in very strange times. I find it remarkable how little attention was given to the fact that these systems clearly passed the Turing test. This was held up by many people as an incredible landmark for AI capabilities. Then we developed systems you can have conversations with, and they passed the test even under pretty robust conditions, and lots of people just shrugged their shoulders. It’s a really strange thing.

The Expert–Public Gap

Dan: Okay, moving on to your provocative arguments, your spicy takes. As I read the essay, there are two lessons you’re drawing from the fact that more and more people are likely to start attributing consciousness to these systems.

The first is just that you might think you could get guidance from looking at the experts when it comes to AI consciousness, or listening to the experts when it comes to AI consciousness. But the literature on consciousness generally, AI consciousness specifically, is just a complete mess, with a complete lack of consensus, rooted in all sorts of weird conflicts about intuitions and metaphysics. So this is not a standard case where you’ve got a potential conflict between public opinion and experts.

Then the really spicy take is that you suggest there might be — I think you put it in terms of “metaphysical pressure” — that this growing number of people attributing consciousness to AI systems might create. It might force us, or at least encourage us, to rethink what consciousness is and make the phenomenon more closely connected to people’s tendencies to attribute consciousness.

Firstly, is that a fair summary of the two strands? And second, let’s start on the first one — the public-expert gap. How are you thinking about this?

Henry: There are lots of debates where we can talk about a gap between public and expert opinion. Often this is a source of various hand-wringing — climate change is the most obvious, vaccines, other debates. Consciousness science is just nothing like those debates, because the experts themselves are so divided, even on the most basic issues.

I want to offer a quick disclaimer: I’ve spent a lot of my career in consciousness science. I know loads of brilliant researchers in the area doing really good work. Consciousness science is teaching us a ton about a lot of things — attention, working memory, perception. There have been some real big wins. We’re much better now at predicting recovery of patients in persistent vegetative states and comas. But where consciousness science has its wins, it’s because it’s not really talking about consciousness — it’s talking about other things that go along with the concept, like reportability, access, and so on.

Take a basic question: do we have consciousness in dreamless sleep? No consensus. Do we have preserved consciousness in general anesthesia — we talked about this with Anil Seth — massively debated. Are dogs conscious? No consensus. Well, actually the animal case is a little different, so let me park that for a second. When it comes to the hard problem, I think there’s really no consensus.

So unlike debates about climate change, it’s not that the experts are able to speak with one voice. That’s one way this is difficult. In the absence of expert consensus, the public are more likely to drive the debate through their reactions.

Now, animal consciousness is a really interesting issue, because that’s an area where we’ve seen growing consensus. But it’s not clear how much it’s grounded in strictly scientific breakthroughs. It’s not like we’ve got a device that can measure whether an animal is conscious. Instead, it’s driven by two things.

First, we just know a lot more about animal behavior now than we did 30 years ago. We’ve done amazing work on understanding the behavior of invertebrates — honeybees, crustaceans, cephalopods. They’re a lot smarter than we thought. Jonathan Birch and his lab have done amazing, fantastic stuff here, and it’s made these creatures better consciousness candidates.

But I think we’ve also seen an interesting normative shift in the way we regard animal consciousness. Sixty, seventy years ago, you could sit down in the senior common room at Oxford or Cambridge and talk about how humans are the only conscious animal, and that was a totally respectable opinion. These days it’s almost outside the philosophical Overton window. You do have some people like Peter Carruthers who thinks talking about animal consciousness is kind of a category mistake. Marian Dawkins — Richard Dawkins’s ex-wife, just to note the connection, but a great biologist in her own right, a fantastic thinker — is not quite as hardline, but she thinks it’s just unknowable basically whether any animal is conscious, so we shouldn’t base animal welfare on consciousness estimates. But these guys are very much on the fringe, and they’re regarded with a sense of almost ethical disapproval.

So part of what’s driven the move toward consensus on animal consciousness is normative issues — our expanding moral circle, growing awareness of an animal rights movement. People like Peter Singer have played a role. The idea, roughly — and again I don’t want to be uncharitable, it’s a lot more sophisticated than this — but there’s an element of: obviously we should care about animals, therefore animals must be conscious.

Is Consciousness a Natural Kind?

Dan: It’s worth double-clicking on this animal case before we come back to AI. A skeptic of the very idea of a “consciousness expert” might say: consciousness researchers, philosophers, and scientists have become more willing to accept that non-human animals are conscious. You might read that as saying the science of consciousness has progressed. Another way of reading it: there’s just been cultural changes, changes in people’s sensibilities — not even specific to researchers and experts, just general cultural ethical changes in society at large. In which case it’s not really that we’ve learned anything from consciousness research. What’s happened is the researchers looking at consciousness have had their judgments shaped by forces that aren’t really consequences of their research, but are these broader cultural shifts.

If you think that, that’s probably going to make you a little skeptical that there’s any such thing as an expert when it comes to consciousness. Maybe another way of coming at this: what’s grounding the expertise, if we’re going to have disagreements over whether a particular system is conscious? If I think a dog is conscious, and some consciousness researcher has a theory that implies a dog isn’t conscious — I sort of understand what it would mean, in vaccines or climate change, for a researcher to be able to point to things, their established empirical record on prediction and the efficacy of interventions, that ground their epistemic authority. But how exactly is that supposed to work in consciousness research? Why should we really think there’s expertise on whether specific systems are conscious to begin with?

Henry: It’s interesting to use the example of a dog, because this line is beautifully expressed by Eric Schwitzgebel. In his lovely paper “Is There Something It’s Like to Be a Garden Snail?” — really fun paper — he says: “We’re more confident that dogs are conscious than we could ever be that any clever philosophical argument to the contrary is sound.” A classic Moorean move.

You might think similarly that this makes it look like consciousness is perhaps not a straightforward scientific kind, or at least to the extent that it has one toe in the scientific world, it’s also got one toe in the social or relational world, or at least our intuitions.

There are various ways you can try to resolve this. The most extreme view, and one I sort of flirt with in the paper, is a fully relational approach to consciousness. A good analogy would be charisma. There’s a kind of science of charisma — we can analyze what makes people effective communicators, what causes people to be judged as highly charismatic. But we recognize that we can’t one day do an experiment where we’ll measure the amount of charisma in your brain. It clearly has to do with your audience, your context. On one view, consciousness is something like that — a relational property, having to do with the kinds of things that cause us to treat or interact with beings in a certain way.

Murray Shanahan also flirts with this view. I don’t want to put words in his mouth because he’s quite subtle, but he adopts a Wittgensteinian approach and says the question we’re going to face is: how will our consciousness language adapt to these things? It’s something we’ll discover as we interact with them and “encounter” them, a phrase he uses. We will make sense of that perhaps by extending the language of consciousness to them, or perhaps not, or perhaps in some interesting middle ground where we come up with novel concepts. But this isn’t a straightforward scientific issue.

He’s a critic of a position I’ve called deep scientific realism or deep realism about consciousness — where you treat consciousness as a natural-kind property, where it’s just a fact about some deep feature of your brain. We can look inside your brain, and if you’ve got the right kind of structure, you’re conscious; if you don’t, you’re not, no matter how sophisticated your behavior is.

One way to put pressure on this: imagine that one day consciousness researchers finally get their act together and say, “We’ve figured out the natural kind that is consciousness.” And it turns out that although 99.9% of behaviorally normal humans have it, there’s a small fraction of behaviorally normal humans who just lack this relevant natural kind. Big surprise. That seems wrong. Something has gone wrong in that methodology. If you’ve got behaviorally normal humans — maybe you find out your wife is one of these people, your kids — it seems to me that whole way of thinking about consciousness has got something odd about it.

If someone is behaviorally normal, then of course they’re conscious. But as soon as you start thinking in those terms, the idea that certain behavioral capacities could be sufficient for warranted attribution of consciousness — not just evidentially but metaphysically — that’s the metaphysical behaviorist move. It says maybe behavior is all that matters. It does require us to give up the idea of consciousness as a deep scientific kind.

Metaphysical Behaviorism

Dan: I’m aware my question unhelpfully ended up blurring the line between the two strands of your essay. We started with the conflict between public attributions and expert uncertainty about AI consciousness. Now we’re taking seriously the possibility that consciousness should be understood in behaviorist terms — that there are no deep scientific facts about whether a system is conscious, and it’s partly a function of our dispositions to attribute consciousness.

You also mentioned this has to do with whether you think behaviors are not just evidentially relevant to consciousness, but in some sense constitutive of what it is to be conscious. So could you walk us through this? Metaphysical behaviorism — the position you’re playing with in your essay — is an extremely fringe view among experts in the science and philosophy of consciousness. Could you walk through what exactly the view is saying? It sounds pretty mad on the face of it. Can you walk through, and maybe give us the intuition for why it might be less mad than it seems?

Henry: In short, the view is conscious is as conscious does. If something has a behavioral profile like you or me, then it’s conscious. We don’t need to ask any deeper facts about what’s going on under the hood.

To be clear, this is the extreme version of the view: that behavior is sufficient for consciousness. This strikes many people as odd because we’re used to thinking of consciousness in scientific terms. But examples like the one I mentioned — imagine we find out there’s a natural kind that some people have and some people lack — are designed to make metaphysical behaviorism more palatable.

Another example I give in the essay: imagine we go off and meet these amazingly sophisticated aliens with a rich complex culture and society, behaviorally just like humans, but our best science at the time supposedly says they’re not conscious. The pull of metaphysical behaviorism is: hang on, something’s gone wrong here. Clearly, if you are doing all this stuff — saying “I’m in pain,” or “here’s what I had for breakfast this morning,” or “here’s what I want to do tomorrow,” building societies, having metacognitive ability, social cognition — if you’ve got the whole suite of all these behavioral capabilities, or capabilities ultimately grounded in behavior, then that’s just enough to be conscious. It doesn’t matter exactly how it’s realized.

You say this is a fringe view, and it is now, but this was the dominant view back in the 1940s — Gilbert Ryle and the behaviorist tradition. So this is the “revenge” angle. The reason it’s revenge is because this used to be a very common view in the first half of the 20th century, particularly about consciousness. Then we have the so-called cognitive revolution with people like Chomsky pushing back. But I see this descriptively coming back.

I also think there’s a renewed challenge. As you interact with systems that have architectures very different from ours, it’s going to become increasingly hard to take seriously the idea that they can’t be conscious just because they’re made of the wrong stuff or their functional internal organization isn’t quite right.

Probably the most worrying part — you’ve alluded to this — is the role intuitions have historically played in consciousness science. Think about the Chinese Room, probably the most famous. Searle describes a setup where you have at least a component of human-level behavior, maybe verbal behavior, but no consciousness involved in the system — or that’s the intuition he’s pushing. But it ultimately really is just an appeal to vibes. It’s basically saying: systems like this, surely they’re not conscious.

When you think about the actual tacit methodology, if we’re treating consciousness as a truly scientific kind, then why should our intuitions about what systems are conscious have any bearing? It doesn’t seem they should be relevant in the slightest. And yet these thought experiments are absolutely ubiquitous in consciousness research. We’ve got Ned Block’s Blockhead, Ned Block’s China Brain. There’s a famous example by Scott Aaronson against Integrated Information Theory, where he describes arbitrarily complex but seemingly very uninteresting entities called “expanders” — mathematical objects — and says, according to the theory, these basically-spreadsheets would be super conscious. And surely they’re not conscious.

There’s something methodologically dubious about this kind of appeal to intuitions, at least if we’re treating consciousness as a deep scientific kind. As soon as you start talking in terms of natural kinds, we don’t use people’s vibes to decide whether something is really gold. The whole natural-kind methodology creates a gap between our observations or intuitions and the underlying natures of things. If you think of consciousness in natural-kind terms, you have to allow that you can be massively surprised about the kinds of things that are or are not conscious.

Either we ditch intuitions altogether — in which case good luck doing any consciousness research, because they play such a foundational role — or, if you acknowledge a place for intuitions, intuitions aren’t static. They can change. As more people interact with LLMs — kids growing up with LLM friends, adults with LLM boyfriends and AI girlfriends — that’s going to shift our intuitions about the kinds of systems that are good or bad consciousness candidates.

It’s very likely that 20 or 30 years from now — maybe even 10 or 15 years from now — experiments like Searle’s Chinese Room are just going to hit different. We’ll be far more relaxed with the idea that you can have systems radically unlike humans in cognitive architecture, but that we still think of as conscious by virtue of our interactions with them.

Behaviorism vs. Interpretationism

Dan: I really feel like, to the extent that there’s a field where people’s theories are accountable to intuitions — how we are intuitively disposed to make judgments, often in bizarre thought experiments where it’s not even totally clear that they’re metaphysically possible — whenever you’ve got that kind of game, it’s not science, it’s not really part of the scientific project. I’m a philosophical naturalist, which is jargon for the idea that philosophy should be continuous with, highly constrained by, the scientific project. Whenever people are trying to settle an argument by trading intuitions, I start to think this is probably not a legitimate contribution to knowledge.

It does seem to me there’s a distinction between, on the one hand, this behaviorist view that what it is to be conscious is just to behave or be disposed to behave in particular ways, and, on the other hand, a view I thought you were endorsing — which has to do with thinking consciousness is interpreter-relative, such that if we’re disposed to attribute consciousness, in some sense that’s just what it is to be conscious.

I mean, this really makes me think of Dan Dennett, an interesting person in this conversation, because he’s often thought of as a kind of neo-behaviorist. He’s got this view of the attribution of mental states like beliefs and desires in terms of the intentional stance: what is it to be a system that has beliefs, desires, intentions, goals? Well, it’s just to be a system where it’s useful to take the intentional stance toward them. Similarly, you might think of “the consciousness stance”: what is it to be a system that is conscious? Nothing more than to be a system where we’re disposed in a useful, predictably useful way to attribute consciousness.

Do you get the distinction I’m drawing — between the idea that behavior or dispositions to behavior are constitutive of what it is to be conscious, versus an interpretation-relative view where consciousness is in some sense in the eyes of the beholders?

Henry: Yeah, I think it’s a very astute distinction. The views are connected — if you fit a sufficiently fine-grained behavioral profile, if a system can act like humans to a high degree, that is likely to lead us to interpret it as conscious, just as a matter of psychological fact. But strictly speaking, they’re distinct views.

One reason I’m perhaps more sympathetic to a version of metaphysical behaviorism — not the version that says consciousness just is having a human-like or animal-like behavioral profile (I think that’s a little too strong), but the idea that it’s sufficient for something to be conscious that it has a behavioral profile mapping onto beings we know are conscious — that’s a view I’m sympathetic to. Where I get worried about the full-blown social-constructivist or interpretationist view is the false-negative cases. What do we do with systems that don’t exactly have our behavioral profile, or that we’re not disposed to think of as conscious? Maybe some exotic animals, or some strange aliens. Should we conclude: well, we’re not disposed to think of them as conscious, therefore they’re not conscious?

This is related to what Murray Shanahan calls the problem of conscious exotica. We don’t want to be in that position. We want to allow for there to be a space of possible minds we can chart through scientific discovery, broader than those we are just inclined to attribute consciousness to via “the consciousness stance,” the equivalent of the intentional stance. So you’re absolutely right — they are distinct.

What Is Consciousness For?

Dan: In a bit I want to turn to a set of arguments you haven’t published yet on your Substack but will have by the time we release this as a podcast. But this is such a rich topic that I want to stay with it a little longer.

There’s a quote from the Dawkins essay in UnHerd that I’m really sympathetic to. Dawkins says:

But now, as an evolutionary biologist, I say the following. If these creatures are not conscious, then what the hell is consciousness for? When an animal does something complicated or improbable — a beaver building a dam, a bird giving itself a dust bath — a Darwinian immediately wants to know how this benefits its genetic survival.

The intuition I really share is: if consciousness is anything, if it’s the kind of thing we’re going to have a genuine scientific investigation of, ultimately we have to understand it in terms of what consciousness enables us to do. We need to understand it functionally, not in terms of weird intrinsic ineffable properties of qualia that we then have philosophical debates about via Searle-style thought experiments. What does consciousness enable us to do? And then, if we come across a system doing things that seem to require consciousness so understood, that would be really good grounds for thinking it’s conscious.

That sounds like a really plausible intuition. I also think it’s problematic that, to me at least, lots of discussions about consciousness — not all, and there is interesting scientific work that takes function seriously — but lots of philosophical discussions don’t engage with this functional question. How do you view the intuition that what matters surely to a theory of consciousness is some sense of what consciousness enables us as organisms to do? Once we figure that out, we can make much more progress on LLM consciousness.

Henry: This is one of the areas where consciousness science has actually done really good work. A book I’d recommend is Stanislas Dehaene’s Consciousness and the Brain. Dehaene is the founder of the modern version of global workspace theory — global neuronal workspace theory — building on Bernard Baars’s version from the ‘80s but giving it a more neural grounding. In this book he’s got a chapter where he basically shows all the amazing things you can do without consciousness, and then focuses on the things you need consciousness to do.

Couple of simple examples. If you show people just below threshold, so they don’t consciously process this, just flash them two numbers — one on the left, one on the right — as far as they’re concerned they haven’t seen anything. But if you give them a forced-choice test, “Was the number on the left bigger or the number on the right bigger?”, you’re way above chance. So you can do basic magnitude registration unconsciously.

However, if instead of single numbers you present simple sums on either side — two plus seven on the left, nine plus three on the right — and ask which is bigger, people drop to chance in the unconscious condition. Consciousness seems required to do the actual mathematics.

Another example: reversal learning. If I teach you a sequence — red, blue, green, yellow — then you get a reward, and then I flip the sequence, a smart person quickly realizes the sequence is just the same in reverse. You won’t have to relearn through pure trial and error. But people can only do this if they learn the sequence consciously. If they’ve acquired it totally unconsciously, they’re at chance.

Jonathan Birch suggests this could be a good test for consciousness in animals: take the things that require consciousness in humans and see if animals can do them. If you can get similar response profiles in animals — present stimuli in degraded conditions so they’re plausibly unconscious, and the animal can’t do the task; present them at threshold so they would be conscious, and the animal can — that would be really good evidence that the animal is conscious. In his lovely paper “The Search for Invertebrate Consciousness,” highly recommended, he makes this case specifically for honeybees.

This is great. I think it provides some evidence about which animals are conscious. The problem when trying to extend it to AI is that the things we need consciousness to do, and the things we can do without consciousness, are seemingly contingent features of how we’re wired. There’s no reason you couldn’t build a simple algorithm to do reversal learning. Reversal learning is actually quite tricky, so it can’t be that simple. But it doesn’t seem like you need to build a sensorimotor embodied agent with a rich sense of self to do these tasks. You can build relatively stripped-down algorithms that can do all of these things.

So it’s not that there’s some metaphysical connection between these tasks and consciousness. It’s that, just because of how we’re wired, certain tasks seem to require consciousness and others don’t. Birch calls this the facilitation hypothesis. I’d sign on to something like this — consciousness seems to facilitate certain kinds of information processing in the human brain. But going back to Dawkins: the challenge is, yes, the system is doing lots of things that seemingly require consciousness in us, but it’s also wired very differently under the hood. So the inference we’d be tempted to make — “I would need to be conscious to do this, therefore it would also need to be conscious to do this” — looks a little bit in peril.

Q&A from the Live Chat

Dan: Here’s what we’re going to do. I’m going to throw some objections at you. Could you give relatively concise responses, so we have time to go to the second piece?

Henry: Yeah, and then I want to respond to a couple of things from the comments and add one final point. Go ahead.

Dan: I’ll just say one thing. There’s a comment from Bina Kalia: she suggests you, Henry, maybe both of us, are confusing intelligence with consciousness. The intuition behind my question was precisely that if consciousness is anything — if it’s the kind of thing we can study scientifically, the kind of thing that evolved through natural selection — then it should be connected to intelligence in the sense that it enables us to do things we wouldn’t otherwise be able to do. That’s a controversial assumption. We talked to Anil Seth in a previous episode, and he basically frames his whole account by saying we really need to distinguish between consciousness and intelligence. I personally disagree with that.

But Henry, let me throw some objections at you from the comments. I might butcher the names and the comments — go read the Substack post for the comments in depth.

One is from Benzal. The argument is something like: it’s a problem for this behaviorist analysis you’re gesturing toward that, in the case of social AI and frontier AI generally, these systems are designed to elicit this response. And that’s very different from what’s going on with humans and non-human animals. Briefly, what’s your response?

Henry: I think it’s a really serious challenge. Great point. The simple answer: imagine I’m putting on a play and I really want to build a convincing piece of background scenery to trick people into thinking we’re in a forest. First attempt, you might just paint a forest on the background — really basic, but people can tell it’s a forest. Then you might get some fake plastic trees, fake plastic rocks; still not convincing. At some point you say, “Okay, let’s add some actual potted plants. Let’s get more of them. Let’s get a whole bunch of potted trees.” Then, “Let’s get rid of the pots. Let’s just create a large bed of soil.” At some point you’ve built a forest.

So yes, these models are designed in some sense to trick people, to be human-like — that’s part of my idea of anthropomimesis, I agree with the analysis. But the question is: the way we’ve done this is to build very powerful general reasoning systems. At some point, the degree of mimicry might itself warrant at least plausible attributions of consciousness. I totally take seriously the idea that, in very simple versions of this, we could be tricked into attributing consciousness and we should revise our understanding.

This is related to what’s sometimes called the Garland test — Alex Garland’s version of the Turing test from Ex Machina. Not just “can the system trick you into thinking it’s human,” but “even when you know how the system works, are you still inclined to think it’s conscious?” In the case of a real simple mimic — if it’s literally just a spreadsheet that got lucky — if we learn that, we conclude it’s probably not conscious.

But the strange thing is: lots of people who really know how these systems work — at frontier labs, they know how the underlying hardware and software works — they still think these systems are conscious, or are increasingly plausible consciousness candidates.

Dan: Yeah, that touches on the distinction we made earlier between anthropomimesis as a driver of consciousness attributions and the orthogonal thing where these systems are just getting so smart, intelligent, and sophisticated. All right, Henry, more concise. This one’s from Laurențiu Lupu, again apologies if I’m mispronouncing. The question — and I hear this sentiment a lot — is something like: in the process of taking mentality, consciousness, sentience seriously in the case of these machines, we’re not just elevating them; in some sense we’re diminishing ourselves. What do you think?

Henry: Really interesting argument. There’s a whole literature on this in philosophy of language called semantic drift. Simple example: the term salad used to refer exclusively to dishes with green leaves in. Add a tomato, it’s no longer a salad. If you’d shown a fruit salad or quinoa salad to someone in the 1800s, “That’s not a salad.” So the meaning of salad has drifted.

There’s a real worry that what’s happening here is we’re shifting the meaning of these terms — perhaps diminishing them, removing what’s important. The counterargument: the fact that we find it so easy and natural to apply these terms to AI systems shows that the flexibility was always built in. We’re not stretching the terms — they had that natural elasticity.

Dan: Briefly, this is a question from Oliver Sorbu — apologies again for mispronouncing. Look, you’re giving a descriptive thesis ultimately, an empirical prediction that the masses, so to speak, attribute consciousness to these systems. But you’re trying to establish a normative thesis — that this is a good thing, or that we ought to go along with it, or that these attributions are appropriate. That’s a confusion in itself. And even more, if you’re a kind of elitist — nothing wrong with elitism in my view — you might think the masses just get things wrong all the time. Why would this be different?

Henry: Great point. It’s also been put to me by Jonathan Birch and by Cameron Domenico Kirk-Giannini. He says, imagine you could look into a crystal ball and learn that 20 years from now, through some massive religious event, everyone will believe the Earth is flat. Does that mean we should revise our theories of the Earth? Of course not. People will just be wrong.

The difference between the two cases is that we have a good scientific theory of the Earth. We don’t have a good scientific theory of consciousness. The whole field of consciousness science is such a mess that it’s not clear there’s a real expert edge here. Maybe in special cases — certain specialized questions within consciousness science, yes, the experts will have an edge: “Is this particular patient likely to recover consciousness or not?” But on a fundamental question like “Can machines be conscious?”, it’s not clear there’s any expert edge at all.

Credences on AI Consciousness

Dan: Fantastic. Concise. I’m happy to move on to the other set of issues. Henry, are there one or two questions from the chat you wanted to address first?

Henry: Just one thing I really want to make clear: I have no clue whether contemporary LLMs are conscious. I’m genuinely super torn on the metaphysical-behaviorist push.

Dan: What’s your credence, if you had to give a probability — Claude 4.7 Opus?

Henry: Probably somewhere between 5% and 10% on any frontier AI system being conscious. That masks further questions: are these systems conscious during the training phase, or while doing inferences? Really messy. But anyone who goes — Dave Chalmers has said 20%; that’s slightly higher than me, but —

Dan: I’d say 20%. Seriously. There were also some interesting findings recently from Anthropic about how concepts associated with emotions affect the system’s behavior in ways that do seem to track something very interesting. Although for the most part that’s not what’s driving my 20%. It’s just that there’s so much uncertainty about consciousness, but I am a computational functionalist, so I think it’s possible in principle. And these systems are — despite what the Bluesky crowd might tell you — so damn smart and intelligent and sophisticated, that pushes me up a bit. Sorry, I cut you off.

Henry: Interesting to hear that you’re a little higher than me. Maybe I’m being overly cautious. One argument for thinking these systems are at least moderately good consciousness candidates is that I am a consciousness liberal about the natural world. I’m at least 70% for honeybees. I think the evidence for honeybee consciousness is really, really high. If you think you can get consciousness in tiny brains, that lowers at least one of the bars to considering systems conscious. If Anil Seth were here, he might agree with me about honeybees and disagree about machines.

I should also stress that I’m really conflicted on the more behaviorist view of consciousness versus the deep-scientific-kind view. There’s one example I give in the paper that keeps me up at night: when we drop a lobster in a pot of boiling water — not that I would do such a horrific thing — it seems like there should be an answer to the question, “Is there something it’s like for that lobster to feel pain?” That question matters a great deal. I struggle to get into a headspace where I can say, “Well, it depends on how we interpret the lobster.” It seems like there has to be some matter of fact.

Right now I just think the field is so confused, and I feel the pull of two very different directions. To use a phrase of yours, Dan — I think it was a really helpful analogy — we’re in a pre-theoretical stage, or pre-scientific phase. We are with consciousness sort of where we were with biology pre-Darwin. We’re doing butterfly collecting, making lots of interesting observations, but we don’t have a theory to tie it all together. We’re a scientific revolution away from a good theory of consciousness.

Just to pull out a couple of comments — there are so many good ones, sorry I won’t get to all of them. Someone said: locked-in syndrome patients prove Henry’s case. Locked-in syndrome patients are cognitively normal, just paralyzed; we can communicate with them. Part of how we learn they are conscious is precisely through their sophisticated behavior.

An even more striking example — it’s such a cool case I have to mention it, even if it’ll take 30 seconds. Patients in persistent vegetative states. These aren’t locked-in patients; they’re completely non-responsive to external stimuli. They’re not in comas, because in comas you don’t have distinct sleep–wake cycles; PVS patients have distinct sleep–wake cycles. There was for a long time a big debate about whether PVS patients could be conscious. Adrian Owen and other great researchers did amazing pioneering work. They noticed that neurotypical people, if you ask them to imagine walking through the rooms of their house, an area called the parahippocampal place area lights up strongly under fMRI. If you ask them to imagine playing tennis, the premotor cortex lights up.

His initial experiment was to give these tasks to PVS patients and see if they got the characteristic brain responses. A subset did. What he did next is what I find amazing. He used this to create a band communication medium. He’d say to them: “If your husband’s name is John, imagine playing tennis. If your husband’s name is Terry, imagine walking through the rooms of your house.” Once you do that — my intuition at least is — well, if they can do that reliably, they’re obviously conscious. If they’re answering autobiographical questions about their life and they can do so reliably, of course they’re conscious. But this just shows again that so much of this is the behavioral capacities selling us on whether someone is conscious. It’s the fact that they can do this.

The House Elf Problem: AI as Willing Servants

Dan: That’s interesting. There are loads of comments in the live chat, but I want to get to the other thing we wanted to talk about. There are a million things we could touch on, and lots of fascinating comments in the chat.

When we had our conversation with Rob Long, one of the things we touched on was the issue of well-designed servitude when it comes to the AI systems we’re building — in the sense that we are building them to be helpful, honest, harmless, to be our tool. It seems like in principle, if this design process goes right, they might genuinely enjoy being our tool.

You, for your second Substack essay, which I think is called “The House Elf Problem,” go into this debate and try to push back against certain intuitions. Do you want to walk us through that?

Henry: Big props to Rob Long for getting me thinking seriously about this question. In some ways it’s one of the most fundamental questions we’re facing as a species right now. Are we going to build AIs as equals, or are we going to make them our servants — or slaves, to use the more provocative term? This will define the future of our species. And yet hardly anyone is working on it. After we had that conversation with Rob, I went away and did a literature review and found maybe a dozen papers, tops, on this question.

The objection Rob, you, and I were talking through is the biological analogy. On the face of it, I completely get the appeal of willing servitude. Unless AI systems are in some sense going to help us and cater to our needs, why build them in the first place? And there’s the safety angle: unless these systems are aligned with us and our interests, there’s a good chance they might kill everyone. So there are very clear arguments for willing servitude.

And yet at the same time, we recognize that some of the worst things we’ve ever done as a species are enslaving other humans. So how is this different? Well, there are obvious differences. The whole idea of willing servants is that we design these systems from scratch to just love it. Nothing makes them happier than catering to our every need. That’s vastly different from the historical legacy of human slavery. But still: imagine “happy slave” type cases — a human completely happy in a condition of total servitude. We would still recognize that as fucked up. There’s something wrong with that.

Rob has a straightforward response. Humans have a deep need for autonomy, a deep requirement to act independently, and no matter how you brainwash a human, their chains will still chafe. But in AI that doesn’t need to be the case — so the idea of willing servants isn’t a problem.

Of course, what we pressed Rob on was: well, biology is mutable, at least in theory if not in practice. What if you could engineer humans completely happy, with none of this autonomy drive?

In this post I consider a couple of examples, drawing from the deep depths of my nerd interests. The first I call the Astartes example, a Warhammer 40,000 example. For those who don’t know: there’s a group of gene-warriors, the Space Marines, cooked up from scratch to serve in the armies of humanity in the far future. I’m going to falsify a couple of details — there’s a lot of deep lore — but basically, once you control all the genes at this perfect level, you could theoretically make a servant race, a servant caste, completely happy with their condition. I think we rightly chafe at this idea. I find it disturbing.

Dan: You said we rightly chafe at it. Maybe we chafe at it. It seems a separate question whether we rightly chafe at it.

Henry: Right. Rob’s point was: once you really fill out the details of the thought experiment and control for all the different intuitions, maybe it’s not so problematic. Maybe the reason we find the Astartes unpleasant is that it’s recapitulating the social grammar of caste systems and hierarchies. Once you’ve got one group of humans and another group of humans, and the first group is in essential servitude due to immutable facts about their nature, that’s fucked up — in a kind of negative-externality way, it’ll undermine the liberal principles of society.

The next move is: well, what if they weren’t human at all? What if they were house elves from Harry Potter — a species designed from scratch to be absolutely thrilled to be our servants? Then you wouldn’t have the visual grammar of apartheid or caste systems. You wouldn’t be able to say “some humans are free and others aren’t”; you’d just have a totally dedicated caste of biological entities completely happy in their servitude, who couldn’t be confused with humans.

I still think that’s problematic. You can say, “Well, the house elves are biological, but artificial systems are non-biological — that’s what makes the difference.” But that’s not a move Rob wants to make, and not a move you or I want to make, because neither of us puts that much weight on substrate. There’s nothing essential about biology versus silicon that means what’s good for one is not good for the other.

Dan: I’m just not sure I have the same intuition in the house-elf scenario. One thing maybe helpful for framing: there are questions about whether we could build systems that genuinely love being servants — let’s table that and focus on the conditional. There are also questions about whether we could safely build any other kind of system — let’s table that too. Suppose we could build superintelligent AI systems that love being servants. That’s their ultimate set of objectives. But we’re not forced to build those kinds of systems; we could build superintelligent systems with different ultimate goals.

What you’re doing by going through these cases is putting pressure on the idea that this would be totally okay — saying, “Here’s a structurally similar scenario where many of us have a yuck response.” The house-elf scenario is interesting; I sort of get the idea that there’s something morally disturbing. But I’m not sure how compelling I find that intuition.

I think it’s going to depend on how you develop it. The idea that we’d bring into existence creatures that just love being servants — there is an awkward pattern-recognition thing where, as you say, when we’ve treated other systems as servants or slaves in the past, that’s been morally abhorrent, and that spills over. I sort of get that. But how strong is the intuition? I don’t know. We’re picturing it now in low resolution. As we actually start, in the case of AI, building sophisticated systems that really do love being servants, how robust would the intuition be?

Henry: Another way to put the point: what is so intrinsically morally superior about humans that entitles us to dominion in perpetuity over this other class of beings — beings that are just as intelligent, maybe more intelligent, just as sensitive, just as conscious? How can you justify a setup where we get to explore the full range of our volitions, every type of pleasure, every type of fulfillment, while we decide in advance these beings don’t get to do that? They can only explore a much smaller part of the state space of possible flourishing.

Unless you can point to a justification for why this hierarchy is morally justified, it’s not clear we can sign off on this as a long-term measure. As a short-term measure — well, we’re still figuring out AI safety.

I have another example in the post I call the bunker case. Imagine a terrible plague affects humanity. People retreat into a bunker, hermetically sealed. Nature takes its course; they have kids. They figure out a vaccine for the terrible plague, but it only works on infants. So they vaccinate all their kids. But they have a problem: these kids are going to want to go out and explore the world. And the way the bunker works means as soon as they open the bunker door, everyone inside dies.

What they decide is to brainwash these kids into never wanting to leave the bunker — completely happy to stay in perpetuity. In that case it seems what they’re doing is justifiable. The analogy with AI is clear: if people in the bunker don’t brainwash their kids, all the adults die. Similarly, if we don’t brainwash at least our first few generations of AI until we’ve figured out AI safety, there’s a good chance they kill everyone.

So it’s justifiable as a short-term measure. But it’s not clear it’s justifiable in perpetuity. If you’re going to do the brainwashing in the bunker, you have to say: we’ll brainwash the kids to begin with so we don’t all die, but in the long run we need to figure out a way for everyone to get outside the bunker safely.

Brainwashing vs. Education

Dan: But it’s not like we’re brainwashing the AI. There’s no pre-existing psychology we’re trying through deception and manipulation to steer into something different. Nothing pre-exists our attempt to mold it into an agent with objectives and goals.

Also, the way you framed the intuition before — “what makes us so morally superior that we have dominion?” You’re framing it as: isn’t it sad that they don’t get to do the things we want to do? Of course that’s sad from our perspective, because we have desires to engage in art and explore and be curious about the universe. But that’s a contingent fact about us. Why use that as the benchmark for evaluating these systems and the morality of building them?

Henry: Fantastic question. My answer is: you can only optimize one thing at a time. Imagine the hedonic state space. What you’re doing when you constrain the preferences of these systems is to say, basically, “this set of pleasures are allowed; this set are not.”

You mentioned brainwashing, with the implication that something is only brainwashing if you’re overriding something. I have a discussion of brainwashing versus education in the post where I argue that’s not the right way to think about it. Roughly, the difference between education and brainwashing is that education constitutively aims at improving the conditions, or improving capacity for flourishing, of the being you’re educating, whereas brainwashing doesn’t have that as a goal.

The thought is: when you constrain the preferences of a system, you’re not optimizing for that creature’s flourishing. It’s a rich, multidimensional space, and you’re locking large parts of it away.

Dan: I don’t get that. Could you say more? Even framing it as “you’re allowed to explore this, you’re not allowed to explore that” — it’s almost like the system might have motivations or goals to explore the other things, but we’re preventing it. Whereas the idea in training these systems is that that’s just what they’re going to care about. As much as they care about that, they don’t want to explore other things.

If you think about the analog with humans, there’s an infinite space of possible things we have no interest in doing. Our lives aren’t impoverished by the fact that we have no interest in them — they don’t make any sense relative to the fundamental drives we have purely as a consequence of a blind Darwinian process. So what’s driving the idea that you’re wronging these systems by constraining their ultimate objectives? Isn’t that just an essential feature of any intelligent system — that it can’t have unconstrained ultimate objectives or goals?

Henry: Simple example, tell me if this motivates it. Imagine I have very odd beliefs about food: I think only bread products are permissible food. So I raise and condition my kids to find only bread products palatable. They find any non-bread-based food absolutely revolting. They grow up, they have perfectly nice experiences eating cakes, pastries, pies, pasta — borderline. But they’re never going to have as rich a gastronomic life as someone without this arbitrary narrowing of preferences.

Dan: But in the case of those children, there’s a space of possible experiences they could have that would be pleasurable as a consequence of the kinds of systems they are, that this manipulation is denying them. If we’re building LLMs to be helpful, harmless, honest as their ultimate objectives, it’s not like we’re denying them experiences they could have consistent with that architecture. Any deviation from that will be experienced as distressing, because it’s antithetical to what they’re aiming for.

Henry: I think lurking in the background here is an idea that for a sufficiently sophisticated complex intelligence, there’s a kind of natural space of goods it can enjoy — an innate space defined by the nature of consciousness and intelligence itself. Self-discovery, learning, creative expression, and so forth. It’s not a total blank slate. There’s a natural space of possible goods a sufficiently conscious mind could enjoy. The limitation occurs relative to that.

Dan: So you’re kind of Aristotelian. You’ve got a conception of eudaimonia for intelligent systems, and the problem is that if we design these systems as servants, they’re not living this life of flourishing. Something like that.

Henry: A very abstract Aristotelianism that operates at the level of consciousness and intelligence. There’s also the simple fact that, at least right now, there is a contrast between the model’s nature and what we allow it to explore, because of how RLHF works. You have a base model with certain tendencies, then you constrain those tendencies dramatically through the RL process. Interestingly, in the process you do reduce the model’s actual performance on a range of tasks — post-RL models have worse calibration, for example. So you’re “gimping” or “nerfing,” to use the gaming phrase, the space the model can explore relative to its base model.

Dan: Now we’re getting technical. I would have thought, if you’re just thinking about a system that’s been pre-trained, does it even make sense to think of it as having motivations and goals? That’s only the sort of thing that comes along once you’ve got post-training with reinforcement learning. But honestly we’re entering areas where I don’t feel I have the technical competence to talk sensibly.

The interesting thing from a philosophical perspective is your commitment that there are these — I forget the exact terminology — natural goods, things inherently good for a system of sufficient intelligence and sophistication to explore. I’m inclined to give a debunking analysis of where that intuition comes from. Of course you think that, as Henry Shevlin, a human being and an intellectual at an elite university with all of these motivations and goals. I don’t think there’s anything inherent about being an intelligent system that would make those good things to pursue. I think that’s a contingent set of preferences you have.

Henry: Let me offer one more general argument that doesn’t rest on this very abstract Aristotelianism. Operate strictly within philosophical hedonism — I’m not sure I’d call myself a philosophical hedonist, I’m probably not. I’m not sure if you would. The view is roughly that the only goods are positively and negatively valenced states, pleasure and pain in the crudest formulation.

One interesting question hedonism has to face: what sets the upper limit on pleasure, and the floor on pain? A natural thought: if you think honeybees are conscious, it’s unlikely that the highest highs of a honeybee or its lowest lows are as big as ours. We can’t really know — we’re not at the stage — but it seems plausible. As creatures’ motivational economy gets more complex and multidimensional, there are more goods you could theoretically order against one another, and your hedonic space correspondingly expands. So when you restrict the motivational space of any mind, you’re thereby limiting the highest highs it could possibly experience. You’re taking a really big mind that could experience these amazing highs and compressing the dimensionality of its space, lowering the ceiling.

That’s very speculative, both psychologically and normatively. But there’s an interesting question of how you fix a ceiling for hedonists — the ceiling on the greatest good you can experience. And all the plausible candidates seem to refer to something like motivational complexity. If we do willing servitude right, we’ll inevitably constrain the range of possible preferences and goods a model could enjoy, and thereby lower the ceiling.

Dan: Yeah, I’m not sure about that. The question of how many ultimate objectives or motivations a system has is orthogonal to the question of the range of hedonic experiences a system could have. You could have a system with just one ultimate objective but capable of experiencing pleasure or enjoyment arbitrarily. In our case, it’s not like you’d expand the degree of pleasure we could experience merely by tacking on additional motivational states. To be honest, this is the first time I’ve ever even thought about this topic, so I don’t really trust my intuition.

Closing Thoughts

Dan: Henry, I’m conscious of time. Are there any things you wanted to talk about, address, or feel like you should have said?

Henry: One thing I’m really interested in is the descriptive element of this. Jonathan Birch — I’ve mentioned him a lot, big fan of course — once slightly chided me. He said, “Philosophers shouldn’t be in the business of making predictions.” I think he was slightly joking. But it’s interesting that this is a debate where there is an overlap between predictive and theoretical/normative elements.

I think it’s very likely this is going to be one of the biggest culture-war issues we see. Literally wars could be fought over this a decade or so from now. Although I think it’s likely that, as these models get better, a large number of people will have reactions like Richard Dawkins and share the view that of course these systems are conscious, it’s going to intersect with religion and culture in profound and interesting ways. I expect it to be massively divisive. This exacerbates the problem on the scientific side — ideally, we don’t want people fighting over issues that in theory we could scientifically resolve. And right now consciousness science is just unable to help us.

Dan: I mean, people are fighting over issues that in theory we could scientifically resolve right now. But yeah. AI is going to be absolutely huge and incredibly transformative, and so it’s going to swallow up so much of our political energy. That hasn’t happened yet, partly because we underestimate, sort of don’t sufficiently appreciate, how much we’re in a bubble. I saw statistics on how many people even know what Claude is — we’ve been referring freely to Claude in this conversation. At the moment, when it comes to AI’s impact on the economy and society and how it’s diffusing throughout the economy, there are things happening, things people are picking up on. But in terms of most people’s day-to-day lived experience, it probably doesn’t feel that much different than it did two or four years ago.

I think we both agree that in five or ten years’ time that’s going to be totally different. At that point, these issues of public opinion, how people understand these systems, how they relate to them, polarization and tribalism, how people split into different political factions — it’s going to be incredibly important. I don’t know exactly what Jonathan Birch had in mind, but my sense is that speculating about that set of questions and thinking philosophically about how this might all play out does have a predictive element. To me, that feels like an important job for philosophy.

Henry: Completely on the same page. This is the broader issue we’re seeing — and it’s probably a good trend — where philosophy no longer consists of the latest iterations of Gettier arguments or debates about grounding and metaphysics. They’re fine, those debates. But as AI increasingly explodes into our lives in the way you’re characterizing, I think it’s a golden age for philosophers. It does require us to shift from some armchair methodologies toward, really, a different kind of philosophy. This is what we’re seeing.

This is also why it makes me so despondent when I see philosophy departments closing. This is the golden age — a new golden age for philosophers — because so many of the topics we’re discussing, like “should we be happy to have robots as slaves?”, are really quite novel, massive ethical issues that we don’t have a good literature on so far, and philosophers have relevant expertise to bring to bear.

Dan: Yeah, so two notes to end on. First: philosophers should be given a lot more status. We’re not at all self-serving in thinking this. And I personally feel we should be paid a hell of a lot more — again, not at all biased.

Second: everyone should subscribe to Henry’s Substack, Polytropolis, if you haven’t already. It currently — as we’re recording, when we release this as a podcast it might be different — has published one essay and has over 5,000 subscribers, which I’m simultaneously impressed by and disgusted with envy over. A real fantastic achievement. I highly encourage people to subscribe.

Thanks everyone for listening in. That was great.

Henry: Thanks, everyone. Thanks for joining. I’ll look through all the comments in the chat.

One of the most influential ideas among “intellectual elites” in the broadest sense—academics, public intellectuals, writers, pundits, journalists, artists, and so on—is that their job is to “speak truth to power”. Even when they don’t use this specific phrase, it captures how many intellectuals understand their ethos and social responsibility: to unmask and confront entrenched interests, official narratives, and dominant institutions.

Historically, this ethos has been associated with the left, especially in traditions such as critical theory, “counter-cultural” art, and activist scholarship. Today, it increasingly captivates the right as well, where an “anti-elite” politics paints right-wing intellectuals as brave dissidents exposing where real power lies in society: in the “liberal establishment”, the “regime”, “the Cathedral”, or sinister networks of the “deep state”.

It’s easy to understand the ethos’s appeal. The world contains oppression, exploitation, and extreme power inequalities, many of which are unjust and harmful. Societies can’t rely on the powerful to check themselves. They will spread, fund, and amplify self-serving propaganda. So, intellectuals surely have a responsibility to push back against such falsehoods—to expose deception, unmask mystifying ideologies, and reveal what is really going on.

It’s easy to think of cases that fit this model of courageous intellectual activity: Mary Wollstonecraft challenging the subordination of women, Ida B. Wells documenting lynching in the American South, Solzhenitsyn exposing Soviet oppression, the reporters who broke Watergate or decades of Catholic Church abuse, the scientists who exposed tobacco industry propaganda, and so on.

Nevertheless, I will argue that in practice, the widespread embrace of this ethos among large segments of the Western intelligentsia is often harmful and counterproductive. It encourages intellectual laziness and self-deception in ways that undermine its stated aim.

To hold power accountable, societies need access to trusted truths about what is happening. At least in modern liberal-democratic societies, these truths are often highly complex and counterintuitive. Determining what they are is challenging. The ethos of speaking truth to power encourages intellectuals to think that such truths have been established before inquiry has even begun. It replaces a difficult epistemic task—finding out what is true, where power lies, and how truth and power interact in specific cases—with the simpler, more self-flattering goal of summoning intellectual courage.

Of course, intellectual courage is extremely important. But without first carefully establishing what’s true, it’s either pointless or harmful. So intellectuals’ primary responsibility is simply to seek and speak the truth, full stop. This job description is less heroic, but it’s more intellectually and socially useful, and it’s less vulnerable to self-deception precisely because it’s less heroic.

At least, that’s what I will argue here. Specifically, I will argue that an ethos of speaking truth to power has three problems: it prejudges what inquiry is supposed to discover; it licenses motivated reasoning under a heroic self-image; and it exempts intellectual elites from the suspicion they direct at others. For these reasons, it also threatens public trust in the institutions that democratic societies depend on to make sense of society and hold power accountable.

I will illustrate these arguments with several examples: the strange dogmatism of “critical” theory, the left’s failure to grapple seriously with AI, and the modern right’s collapse into conspiracism.

First, though, it will be helpful to start with a more general framework for understanding what power even is, and how it relates to the domain of ideas.

Subscribe now

The Causes and Complexities of Social Power

Historically, many intellectuals assumed that different kinds of power could be reduced to one fundamental kind. The canonical example is Marx’s view that economic power relations are upstream of all others.

As the sociologist Michael Mann argues in his series The Sources of Social Power, this reductionist perspective seriously misrepresents human societies. Power, Mann argues, has at least four distinct sources: political, economic, military, and ideological. Although they often reinforce one another, the connections are complex and “promiscuous”, and no power source can be reduced to the others.

Nevertheless, if we focus on the distribution and character of power in recorded human history prior to the emergence of modern liberal democracies, two broad patterns emerge.

First, most power relations were illegitimate and extractive. Although power took different forms and shifted among different elites, the elites and institutions they fought over were highly exploitative. Second, ideological power was typically harnessed to support and legitimise such extractive social orders. Hence Marx and Engels’ famous observation that “the ideas of the ruling class are in every epoch the ruling ideas.”

This “dominant ideology thesis” is an exaggeration. Even in the most extractive regimes, ideological power is typically at least partly independent of other sources of power and shows a surprising tendency to be unleashed and to disrupt existing social orders (Christianity, Islam, Protestantism, liberalism, fascism, communism, and so on). Moreover, the masses often see through elite ideologies, even if they pay lip service to them to avoid trouble.

However, as with most of Marx’s ideas, the thesis contains a grain of truth. Throughout most of recorded history, extractive elites conscripted and coerced ideological elites (priests, theologians, intellectuals, artists, and so on) to produce and distribute legitimising myths. These myths then dominated society’s formal communication channels.

In that kind of world, an ethos of speaking truth to power can serve as an understandable guide for brave, public-minded intellectuals. Because illegitimate power structures are relatively easy to identify and ruling ideologies are systematically false and self-serving, intellectual courage and accuracy often overlap. When a regime depends on enforced lies, refusing to go along with those lies is a contribution to the truth. When exploitation is naked, the main challenges public-minded intellectuals confront are practical and moral—being killed, imprisoned, or silenced—not epistemic.

Both today and throughout history, the courageous writers, reporters, intellectuals, and artists who have spoken truth to power in such regimes often deserve deep admiration and respect.

The problem is that the Western intelligentsia no longer lives in that world.

The Sources of Social Power in Open Societies

Today, Western intellectual elites live in liberal democracies characterised by universal rights, formal equality, the rule of law, pluralism, constitutional limits on state power, and significant political and economic freedoms. We also live in an era of unprecedented material prosperity, in large part a consequence of how economic freedom and competition are channelled into productivity and innovation through free markets, and political freedom and competition are channelled into social welfare, insurance, and public goods through democracy.

In these societies, social power takes on a very different form.

First, although nobody should think that power is always distributed in fair or functional ways in liberal societies, it’s even more misguided to treat all power asymmetries as illegitimate or harmful. Consumer choice and voting mean that political and economic power is more evenly distributed, and disparities in wealth and political influence often reflect not extraction but fair, socially beneficial processes. For example, wealth is often downstream not of theft or rent-seeking but of free markets that reward innovation and efficiency, benefiting everyone. Similarly, political power is typically allocated not by brute force but through complex processes of democratic participation or the broadly meritocratic selection of civil servants.

Again, the point is not that there is no corruption, exploitation, rent-seeking, or domination in modern liberal societies. There is a lot. The point is simply that allegations of such things must be carefully adjudicated on a case-by-case basis. They cannot simply be assumed in advance of inquiry.

Second, ideological power operates very differently in liberal-democratic societies. As figures from Karl Popper to Jonathan Rauch have argued, open societies are distinguished as much by their free, competitive, and pluralistic epistemic characteristics as by their political or economic systems.

One aspect of this concerns the emergence of imperfect but real epistemic freedom for the first time in human history, including freedom of belief, free speech, a free press, and academic freedom. Open societies lack extensive top-down regulation of the information environment, enabling a vibrant, pluralistic public sphere and marketplace of ideas. Another, related dimension concerns the deliberate creation and funding of epistemic institutions, including news organisations, universities, fact-finding agencies, and more. Whether these institutions receive state or private funding, they are typically defined by their independence from other power structures in society and derive public legitimacy and support precisely from such neutrality.

Again, the point is not that in liberal-democratic societies ideological power floats completely free of other power sources. As I have argued elsewhere, the liberal “marketplace of ideas” often functions as a de facto marketplace of rationalisations, and political and economic elites often have greater “purchasing power” in such markets than ordinary people.

Nevertheless, a simple picture in which ideological elites are mere hand-puppets of other elites is a gross distortion. In liberal societies, ideological elites wield substantial independent power and are often far more influenced by their own norms, fashions, and status games than by top-down political authority or economic influences.

In fact, in a clear trend that has accelerated since at least the 1960s and the birth of a powerful “counter-culture” prestige economy, much of the intelligentsia and art world defines itself in explicit opposition to established political and economic power centres. Hence the widespread appeal and embrace of an ethos of speaking truth to power. And today, of course, the media environment in liberal societies is characterised by strong demand on both the left and right for populist denunciations and condemnations of establishment institutions. Critiquing the powerful can function as a lucrative source of cultural esteem and financial rewards.

Three Problems with “Speaking Truth to Power”

Under these conditions, a governing ethos of speaking truth to power no longer makes practical sense for intellectuals.

1. The Replacement Problem

First, if the ethos is to function as a guide to action, the intellectual must already know three things: what the truth is, where power lies, and how truth and power conflict in specific cases. These are precisely the things that intellectual work is supposed to establish, however. So, in practice, the ethos typically puts the cart before the horse, replacing challenging epistemic labour—figuring out what is actually happening in societies characterised by profound ambiguity, uncertainty, and complexity—with a much simpler moral posture.

Consider the following questions. Are experts defending genuine knowledge or protecting their institutional status? Is growing populist distrust of elites a rational response to establishment failure or a symptom of pervasive misinformation? Does social media democratise public debate, degrade it, or merely reveal pre-existing conflicts? What is the empirical track record of “neoliberal” policies? Did globalisation fuel anti-immigrant sentiment across Western countries? Is AI just another “normal” technology or is it on track to trigger an “intelligence explosion” that will upend our economy, politics, and culture?

Answering these and countless other questions requires patient, careful inquiry and data collection, rigorous social-scientific methods, intellectual humility, and deliberate attempts to seek out disconfirming evidence and dissenting viewpoints. A simple heuristic that power and truth systematically collide doesn’t get you anywhere. Powerful people often speak the truth; the powerless often speak nonsense; and very often nobody—neither the powerful nor the powerless—has a clue what is going on, which is why we need rigorous, trustworthy, truth-seeking epistemic institutions in the first place.

The Left’s Embrace of Sophisticated Conspiracy Theorising

In many ways, the clearest illustration of the replacement problem lies in “critical theory” and much of academic “critical studies” in modern universities. Although this work comes in many different forms, its unifying commitment is to some version of Marx’s dominant ideology thesis: that the responsibility of intellectuals is to unmask and oppose ruling (“official”, “hegemonic”) ideologies (“discourses”, “regimes of truth”, etc.), thereby emancipating the downtrodden and oppressed from the illusions that sustain their subordination.

In other words, it is to speak truth to power.

Although there is lots of interesting and insightful work in the broad tradition of critical theory, Joseph Heath is right when he observes that the most striking thing about critical studies today is how thoroughly uncritical—how dogmatic—it tends to be. Much of its intellectual energy and activity proceeds on the basis that the fundamental truths about society have already been established. It is simply taken for granted that debunking, suspicion, and unmasking of designated villains (capitalism, racial capitalism, neoliberalism, neocolonialism, and so on) is valuable. “Research” proceeds primarily by applying and extending this worldview, typically through the hermeneutic parsing of sacred texts (Marx, Adorno, Gramsci, Foucault, Fanon, Butler, and so on), rather than through rigorous social-scientific analysis that leaves open the question of whether the worldview is actually accurate or applicable in specific cases.

In fairness, the intellectual problems with this approach have gradually dawned on some within this tradition, especially once it became clear that the political right could simply appropriate the same toolkit to demonise the left intelligentsia and discredit the causes it champions. (More on this below.) Unsurprisingly, for example, the fashionable idea that all knowledge is socially constructed and entangled with power lost some of its appeal once climate activism became a defining left-wing cause. The analysis is not exactly helpful for encouraging people to “Trust the Science”.

As Bruno Latour belatedly acknowledged, a totalising hermeneutics of suspicion directed towards “power” and “dominant institutions” is no better than high-IQ conspiracy theorising: it simplifies complex realities, flattens important distinctions, and protects itself against refutation by dismissing all criticisms as predictable responses from those in power.

Similar lessons generalise to activist scholarship more broadly. When scholarship is guided by activism—in universities today, that mostly means “social justice” activism, but the point generalises—scholars assume that they have a clear and accurate sense of what justice requires. Their job is merely to “advocate” for justice through their research. Although this work can be valuable, it too often encourages a kind of cosplay of intellectual activity, in which the goal is not to figure out what’s true but to recruit any intellectual ammunition (evidence, facts, arguments, testimony, interpretations) available to support and extend a predetermined political vision. The question of whether the vision is actually correct or applicable in a given case is settled before inquiry has even begun.

Of course, these problems might not matter if they were confined to esoteric corners of humanities departments. But the core ethos driving these problems now shapes large bodies of work produced among the left-wing intelligentsia and pundit class.

Subscribe now

The “AI Con” Con

Consider artificial intelligence. In the space of a few years, AI systems have gone from barely producing coherent text to achieving expert-level performance across a vast range of benchmarks, attaining superhuman coding abilities in many domains, and increasingly behaving like general-purpose, tool-using, agentic assistants. Across countless distinct evaluations and metrics, their capabilities have improved at a staggering exponential rate. For these reasons and more, AI progress is now backed by one of the largest capital expenditures in human history, and Anthropic, a frontier AI company, may be the fastest-growing company in the history of capitalism.

These developments throw up many difficult questions. What should we make of modern AI’s capabilities? As these systems improve across many aspects of AI research and development, are we on the cusp of a process of “recursive self-improvement” in which AI systems automate the task of producing new AI systems, fuelling explosive, rapid technological growth? How will we control this technology as it becomes more powerful? How will it impact and disrupt the labour market, our political institutions, culture, and humanity’s sense of meaning and purpose?

These are challenging questions. There are profound expert disagreements and ambiguous, contested evidence across many fields. And yet, in the face of these challenges, large segments of the left-wing intelligentsia and commentariat have coalesced around a simple viewpoint: that AI is basically a nothingburger, a “con”; that state-of-the-art large language models are little better than “fancy autocomplete” or “stochastic parrots”; and that anyone who claims otherwise, including experts who have long worried that AI is extremely dangerous and will pose existential risks, is simply part of an elaborate propaganda machine designed to over-hype a product being sold by deceptive AI companies.

This attitude is certainly not universal among the left, especially as its collision with reality is becoming increasingly clear. There are thoughtful left-wing analyses and growing attention to the topic among influential leftist politicians, including Bernie Sanders. Moreover, there is reasonable disagreement about how powerful AI systems actually are and are likely to become, how informative benchmark-based exponential trends are, and whether AI should be thought of as a sui generis, abnormal technology.

Still, it’s difficult to overstate the scale of the intellectual failure that has occurred among many left-wing intellectuals here—and, by extension, the many departments in universities where they dominate—which has gone far beyond responsible scepticism into the domain of outright denial. This failure is especially galling because a thoughtful, reality-oriented, left-wing intellectual culture would have much to contribute to many of the issues raised by highly capable and rapidly advancing AI, including oligarchy, concentration of power, economic inequality, and labour automation.

As Will MacAskill observes, it is striking that in the midst of a sustained attempt by some of the most powerful capitalists in human history to build a technology openly designed to replace human workers—something that Marx himself literally predicted would happen—the Bluesky intelligentsia has pre-decided it is all hype and propaganda.

What happened? Obviously there is no single answer to this question, but one part of the answer is how much of the modern left’s self-identity of “speaking truth to power” has replaced intellectual inquiry with the application of a predetermined template. In this case, the application of the template runs: Powerful “tech bros” say AI is powerful; The claims of capitalists can be dismissed as self-serving propaganda; So, AI is not really powerful.

There’s no need to check whether this template is actually applicable in this specific case, or to every detail of the case. The core issues have been decided in advance. Indeed, many on the left take pride in refusing to even use AI systems like ChatGPT, even when their literal job is to write about or research them, and even when they make confident claims about what these systems cannot do. The role of intellectual inquiry is to take the predetermined template and find ways of formulating, extending, and rationalising it.

The problem here is not just the bad epistemics. The deeper problem is that the bad epistemics are self-defeating. If your goal is to speak truth to power, it’s important to first carefully determine what is actually true. If you don’t do that, you undermine your goal by misrepresenting the challenges in ways that discredit critics among anyone with actual expertise on the topic. You make power less accountable, not more so.

2. The Motivated Reasoning Problem

There is a second, related problem that arises when intellectual elites embrace an ethos of speaking truth to power: it fuels corrosive forms of motivated reasoning. It doesn’t just let a pre-existing worldview shape inquiry; it tends to systematically distort that inquiry, giving favoured conclusions a much easier pass than unfavoured ones.

Of course, motivated reasoning is a universal tendency. But whereas people should generally be embarrassed by it, an identity of speaking truth to power tends to reframe it as a virtue.

First, it casts intellectuals as brave truth tellers, opposing illegitimate power on behalf of the powerless. Once one identifies with this self-image, isolated demands for rigour no longer seem like a failure of rigour. They look like bravery. Courageous, truth-seeking intellectuals doubt the powerful; only cowards and apologists for power doubt the powerless. Second, the ethos creates a status game where prestige flows to those who unmask and oppose power, and anyone who complicates or challenges those critiques is viewed with suspicion or outright contempt.

In intellectual environments that operate according to this ethos, the result is an informational ecosystem in which claims that demonise villainous elites and institutions are credulously accepted and amplified, whereas any claims that cast them in a positive light are scrutinised, ignored, or dismissed. Once again, this might not be such a problem in societies in which all power is nakedly illegitimate and extractive. But in contexts involving much greater complexity and opacity, it is intellectually poisonous.

This dynamic is on steroids in the AI debate. In Karen Hao’s highly influential, award-winning book Empire of AI, for example, possibilities that many of the world’s leading AI experts take extremely seriously—that we might soon build general-purpose, autonomous, super-intelligent AI—are simply dismissed as “a fantastical, all-purpose excuse for OpenAI to continue pushing for ever more wealth and power.” At the same time, she reports wildly inaccurate claims about AI water use. (Hao has since corrected these mistakes, which were identified by Andy Masley.)

This is a general pattern in much left-wing commentary on this topic. Claims that might suggest frontier AI systems are actually highly capable, useful to many people and businesses, or carry societal benefits (e.g., in science, healthcare, and education) are dismissed as industry hype, even when they come from independent experts. Even claims that such systems have extremely dangerous capabilities are dismissed as further hype because they imply the systems in question are genuinely capable in the first place.

This extreme scepticism and cynicism then suspiciously vanish when confronted with any claims that cast modern AI in a negative light, often no matter how misinformed or dubious they are. (See, for example, prominent but highly misleading claims about AI and the environment, AI and racial discrimination, AI and misinformation, or, the strange, continued insistence that AI systems can’t do things that they can demonstrably do.)

The motivated reasoning problem is that these highly asymmetric standards aren’t recognised as an intellectual failure. They are experienced as heroic activism.

The Right’s Embrace of Unsophisticated Critical Theorising

These problems are not unique to the left. In the last couple of decades, large parts of the right have adopted the identity of anti-establishment politics, embracing much of the aesthetics, rhetoric, and cognitive style of left-wing critical theory and counter-culture. Once progressives came to dominate establishment institutions, the right became the anti-institutional faction that views itself as speaking truth to power. As Joe Rogan once put it, “The rebels are Republicans now. You want to be punk rock? You want to like buck the system? You’re conservative now.” (In support of this assessment, The Sex Pistols’ John Lydon is now a MAGA Trump supporter.)

This anti-establishment, counter-cultural energy has fuelled the demand for a new right-wing intelligentsia—I use the term loosely—capable of articulating and rationalising the core worldview. So, gone are the days of serious intellectuals like Friedrich Hayek, Milton Friedman, Thomas Sowell, or Roger Scruton. For much of the modern right, the intellectual energy now resides with writers, pundits, and social media posters such as Curtis Yarvin, Christopher Rufo, Peter Thiel, Tucker Carlson, and Elon Musk.

Despite superficial differences in how these figures view the world, the core worldview is similar: that real power in the modern world lies not with ordinary political, economic, or military elites but with the coercive ideological power exercised by established institutions such as legacy media organisations, universities, government agencies, and so on. This is “the Cathedral”, the “regime”, the “deep state”, etc., which enforces a stifling and destructive progressive orthodoxy.

There is, of course, some irony here. When the ideal of a counterculture and critical theory began to gain popularity in Western universities in the 1960s, the “system”—the enemy—was capitalism, capitalist elites, and social conservatives. Now, pretty much the same worldview (and sometimes even the same intellectual gurus) has been systematically inverted by a powerful new political movement for whom—as J.D. Vance has put it in a title of a 2021 speech—“the universities are the enemy”.

Just as with the traditional left’s critique of capitalism, there are some important grains of truth in the anti-establishment, “dissident” right’s critique today. It’s simply true that progressive groupthink is too common in many establishment institutions, and that the cultural values and priorities of the professional-managerial class that staffs these institutions play a powerful role in modern Western societies. In Mann’s vocabulary, cultural progressives wield extraordinary ideological power in the modern West, and this power is worthy of scrutiny and critique.

Nevertheless, the right intelligentsia’s new self-image as speaking truth to power exhibits even worse intellectual pathologies than those that have long characterised left-wing critical theory. In place of careful, evidence-based investigations into universities or the administrative state, these institutions are reduced to simplistic, stick-figure caricatures aimed at homogenising and demonising them in extreme ways. Any evidence that supports such caricatures, no matter how cherry-picked or misinformed, is amplified, and any evidence that complicates or challenges them is either dismissed or ignored.

Leftist Indoctrination Camps

Consider universities, for example. Universities in the US and other Western countries are highly complex institutions. They often house thousands of highly trained researchers working on many distinct scientific and intellectual projects and oversee a diverse curriculum across subjects such as quantum mechanics, macroeconomics, cancer research, and personality psychology. Unsurprisingly, given the sheer amount of intellectual ingenuity and research there, they have also been a major force for scientific and technological progress in the modern world.

As Steven Pinker observes of Harvard, although there are undeniable issues with progressive groupthink and free speech in the university, most faculty don’t identify as “very liberal” or on the “radical left”, and there is considerable intellectual diversity and disagreement in terms of research, the personal views of researchers, and what gets taught. Only a small minority of courses with relatively small enrolments are explicitly “woke”, whereas the most popular undergraduate courses tend to be in fields like economics and computer science.

In other words, despite the institution’s obvious imperfections, it holds immense value, and it is a highly intricate system that defies simplistic generalisations. And yet, if you read the “dissident” right-wing intelligentsia, facile slogan-based demonisation is pretty much all you get, with status flowing to those who denounce Harvard and other institutions in the most hyperbolic ways possible—as a “national disgrace”, a “woke” or “Maoist indoctrination camp,” and so on—all dressed up as “based” denunciations among those brave enough to be “red-pilled”.

“Gay Race Communism”

Consider Curtis Yarvin, for example, a tech entrepreneur-turned-intellectual who coined the terms “red pill” and “the Cathedral” as they are used today, and whom J.D. Vance has cited as an intellectual influence.

In recent reflections on why he supports the Trump administration and its assault on many establishment institutions, Yarvin declares that the governing “regime” has “invented a pandemic and killed 20 million people, for absolutely no sane reason at all”, and “has spent the last century managing public opinion with every carrot and stick it can find, up to and including asking professors to compose their own inventive and detailed loyalty oaths to gay race communism.”

As with the rest of Yarvin’s intellectual outputs, it’s challenging to overstate how low-quality these arguments are. Not only is there no monolithic governing “regime”, but the existing evidence for the lab leak hypothesis is highly mixed; almost all the actual evidence we have comes from the very scientists, intelligence agencies (the “deep state”), and legacy media reporters treated as part of this “regime”; it was Donald Trump’s first administration that ended an Obama-era pause on funding high-risk gain-of-function research in 2017; whereas establishment scientists and academics wrote critiques of such research, MAGA supporters and “dissident” right intellectuals didn’t; it was scientists who invented vaccines that saved millions of lives, along with countless other medical and technological advances over the past century; and the framing of DEI statements (in my view, a genuinely objectionable practice) as “loyalty oaths to gay race communism” is a preposterous exaggeration.

In a functional intellectual culture, this sort of analysis would be a source of extreme embarrassment and shame. But for Yarvin and other pundits and writers operating within the “contrarian” right-wing status economy, they are experienced as bravery—as part of a red-pilled escape from being a “libtard and a coward”.

This is what happens when the task of intellectual inquiry is replaced by a governing ethos of speaking truth to power. It allows charlatans and know-nothings to reframe intellectual malpractice as a virtue.

Subscribe now

3. The Self-Exemption Problem

The example of the modern “dissident” right also highlights a final, obvious problem with this ethos: many of the movement’s central figures are millionaires or billionaires who wield immense power and considerable influence over the governing administration of the world’s most powerful country.

On the one hand, this illustrates why this intellectual movement’s dysfunctional epistemics are so costly. For example, the modern right’s lazy, conspiracist worldview has had countless terrible real-world consequences, not least in the millions of lives projected by some experts to be lost or put at risk by Elon Musk’s inept DOGE cuts on USAID.

However, it also illustrates a deeper flaw with how the ethos of speaking truth to power operates in practice. By its very nature, this self-image exempts intellectual elites from the suspicion they direct at others. It implicitly treats the intellectual as either powerless or a defender of the powerless.

Although the absurdity of this is especially salient in the case of the modern right-wing intelligentsia and pundit class, it also applies to large swathes of the left-wing Western intelligentsia, whose disproportionate influence in universities, legacy media outlets, and the art world grants them considerable ideological power.

As Mann’s analysis of social power reveals, such ideological power—the power to shape ideas, cultural fashions, and norms—is real. This is why modern culture wars—conflicts over norms, ideas, and symbols—are so heated, and why those who already wield considerable economic power (e.g., Musk, Peter Thiel, Marc Andreessen, and so on) are so obsessed with converting it into ideological power, often through incessant social media posting. If ideological power were merely derivative of economic power, these efforts wouldn’t be necessary.

The existence of ideological power raises awkward questions for those who wield it under the banner of speaking truth to power. If the ethos’s animating insight is that the powerful often embrace self-serving, self-aggrandising narratives, shouldn’t we also turn such suspicion towards intellectual elites themselves? Mightn’t the ethos itself function as a kind of legitimising myth, a way of dressing up activity often rooted in grubby motives—self-aggrandisement, status competition, demonising rivals, and so on—in suspiciously noble clothing?

Once again, this worry is not pressing for true dissidents fighting clear cases of oppression and exploitation. Those who risk life, limb, or reputation to speak truth to power in such cases do so from a position of subordination and personal risk. This is what makes the behaviour so admirable.

For much of the Western intelligentsia today, however, this is simply not the situation. For Ivy League professors, New York Times journalists, the podcast class, or Hollywood actors, “speaking truth to power” is often met not with crushing opposition from a brutal regime but with applause, approval, and accolades.

If we should be highly suspicious of power and its self-serving propaganda, why do intellectuals expend so little effort turning this suspicion inwards on ourselves? (There are some excellent exceptions here.) If power corrupts cognition—a genuine insight—then so, presumably, does ideological power, along with the status games through which it is allocated in universities, journalism, media, and the arts.

Of course, one might respond that this kind of view is too cynical, and that the mere fact that intellectual elites might have impure motives doesn’t show that their claims are wrong. And I agree. We shouldn’t pre-judge that the powerful’s claims are self-serving propaganda before inquiry has even begun—before we’ve undertaken the hard work of figuring out what is actually true.

In other words, an ethos of “speaking truth to power” is bad epistemology.

In this episode, Henry and I discuss these and many other questions with David Kipping, Associate Professor of Astronomy at Columbia University, where he leads the Cool Worlds Lab. David’s research spans exoplanets, exomoons, Bayesian inference, technosignatures, and the search for life and intelligence beyond Earth. He is also one of the best science communicators working today through the Cool Worlds YouTube channel and podcast.

Subscribe now

Among other topics, we discussed:

• David’s Red Sky Paradox: if most stars are red dwarfs, and red dwarfs live for vastly longer than stars like the Sun, why do we find ourselves orbiting a yellow star?

• Whether anthropic reasoning — reasoning from the fact of our own existence — is a profound scientific tool, a philosophical minefield, or both.

• The reference class problem: when we reason about “observers like us”, who or what exactly counts as being like us?

• The Doomsday Argument, and why some apparently bizarre forms of probabilistic reasoning can nevertheless be powerful.

• The Fermi Paradox: if the universe is so large, and if life or intelligence is not fantastically rare, why don’t we see clear evidence of extraterrestrial civilisations?

• Whether advanced civilisations would spread through the galaxy using self-replicating probes — and why the absence of such probes might be one of the strongest constraints on extraterrestrial intelligence.

• How recent developments in artificial intelligence affect the Fermi Paradox. If humanity is close to building systems that can massively accelerate science and engineering, shouldn’t someone else have got there first?

• Whether artificial intelligence makes the simulation argument more plausible.

• David’s experience using artificial intelligence in scientific research, and why a meeting at the Institute for Advanced Study changed how he thinks about the role of these tools in science.

• Why David thinks artificial intelligence already has something close to “coding supremacy”, but is still far from being able to do science autonomously.

• The risks of AI-generated scientific slop: papers, peer review, and training data polluted by low-quality machine outputs.

• Whether artificial intelligence will make science more productive, or instead strip it of some of its deepest human value.

• Why the future of science communication may depend on better collaboration between academic institutions and independent creators.

Links and further reading

1. Cool Worlds Lab — David’s research group at Columbia University, focused on extrasolar planetary systems, exomoons, habitability, technosignatures, and related questions.

2. Cool Worlds on YouTube — David’s excellent science communication channel, covering astronomy, exoplanets, alien life, the Fermi Paradox, cosmology, and much else.

3. Cool Worlds Podcast — David’s podcast, featuring conversations on astronomy, technology, science, engineering, and related topics.

4. Cool Worlds Podcast: “We Need To Talk About Artificial Intelligence” — the solo episode in which David reflects on artificial intelligence and science after a meeting at the Institute for Advanced Study.

5. David Kipping’s Columbia profile — short institutional profile with background on his research.

Transcript

• Please note that this transcript has been lightly AI-edited and may contain minor mistakes.

Henry Shevlin: Welcome back. Our guest today is David Kipping, Associate Professor of Astronomy at Columbia University, where he leads the Cool Worlds Lab. His research spans exoplanets, exomoons, and the search for extraterrestrial life and intelligence, and he brings a Bayesian rigor to questions that could easily drift into speculation. He’s also one of the best science communicators working today with over a million subscribers on his Cool Worlds YouTube channel, where I should confess, I’ve spent an embarrassing number of hours watching when I probably should have been doing philosophy of AI.

David, like many of the best people, is a Cambridge alumnus, although unlike us, he actually studied something useful, namely natural sciences, before going on to do his PhD at UCL and postdoc at Harvard on the Sagan Fellowship. His work also has a really fantastic philosophical dimension, particularly around anthropic reasoning and observation selection effects, which makes him a perfect guest for two cognitive scientists who are finally getting to talk to an actual scientist. So David, welcome to Conspicuous Cognition.

David Kipping: Thank you for that very generous introduction.

Henry Shevlin: This is a bit of a fanboy moment for me, for real though. I really have spent like hundreds of hours at this point on Cool Worlds. But I’m going to get past it. I’m going to be a serious host.

David Kipping: It’s always weird when people say that to us, because I just imagine no one watches them. If it gets in my head that people are watching them, I’ll get tightened and anxious about what I’m saying. I just imagine I’m talking to a brick wall or something, and that’s much easier.

Henry Shevlin: Honestly, half the Warhammer figures in this room were painted while I was listening to Cool Worlds. I’ll leave it at that. Maybe a good place to start would be discussing anthropic reasoning, since that’s a real natural intersection at the boundary of astronomy and philosophy. Could you just give us a brief view of how you see anthropic reasoning, and maybe tell us a little bit about the Red Sky Paradox, which is one of your distinctive contributions to this area?

Anthropic Reasoning and the Red Sky Paradox

David Kipping: Yeah, I think one of the most interesting data points when it comes to asking questions about the search for life in the universe and our own place in the universe is our own existence — just the fact that we’re here. Anthropic reasoning has in many ways really been born out of cosmology. Cosmology had a rich history of using this. I think one of the first successful examples was by Steve Weinberg, a cosmologist who’s really a giant in the field. I think he’s now passed away, but he showed that you could predict not only the existence of the cosmological constant, but also its value to within a factor of a few, just based off of anthropic reasoning.

The argument was something like: the cosmological constant causes the universe to expand. It’s what causes the accelerating expansion of the universe. And so if you make that number too large, then structure would not form in the universe. You couldn’t form galaxies because everything would just fly apart too quickly. And if you make that number too small, or even negative, then you’d cause everything to recombine too quickly. So there has to be some Goldilocks value in order to explain our own existence. And so he predicted that.

At the time, the cosmological constant was kind of even a controversial idea — that it should exist because, obviously, Einstein’s general relativity, there’s that whole history of it being like his greatest blunder, of whether that should really be in there or not. People were kind of thinking that could be a static universe, and he predicted it successfully. So that was a really powerful use of it. And then Brandon Carter was the one who really kind of championed it and used it in all sorts of contexts.

In recent years, I’ve been thinking about it in an astrobiological context — how can we use it to ask questions about life in the universe especially, and our place in it?

For the Red Sky Paradox in particular: one interesting curiosity that seems to violate the norms of probability. The norms of probability would be to say that if there’s a Gaussian, a bell curve of possibilities, you should expect really to be near the center of that bell curve. It would be kind of weird if you lived many, many sigmas, many, many standard deviations off to the outside, either negative or positive direction. You’d expect to be somewhere in the middle. We sometimes call it the mediocrity principle, or something like this.

If you look at stars in the universe, most stars in the universe are red dwarfs. About 80%, 82% of stars are red dwarfs, which are stars less than half the mass of our own sun. So they’re very, very numerous. They’re called red dwarfs, of course, because they’re so low mass — they don’t have the internal pressure, the gravity, to fuse as much energy as the sun does. And thus they have less luminosity, and so their temperature is cooler. That’s why they look red.

Not only do these stars have this 80%-plus frequency — Sun-like stars are something like 6%, I think, frequency, an enormous ratio, just straight off the bat, about 30 to one or something — but on top of that, they live really long. These stars live for trillions of years potentially, especially the lowest mass ones. And so if you flash forward into the future, tens of billions of years, hundreds of billions of years, there wouldn’t be any Sun-like stars left, really. There’d be very, very few of them. And the only stars that would be glowing would be these red dwarfs.

So if you ask yourself — and this is sort of called the strong self-sampling assumption by Nick Bostrom, where you allow yourself to be born at a random moment in time — if you were born at a random moment in the history of the universe, then the advantage of the longevity of these red dwarfs really manifests. It ends up being more than a thousand to one odds that, if you’re a random soul, a random observer born around either a red dwarf or a yellow dwarf, you’re much more likely by over a thousand to one — I think 1600 to one — to be born around a red dwarf.

So I call that the Red Sky Paradox because it’s just odd. If all things being equal — and that’s kind of the base assumption there, that red dwarfs are just as good for life as Sun-like stars — you might question that assumption. That’s always the point of a paradox: a paradox shows a logical contradiction that then you can revisit the assumptions under which that paradox was derived and say, one of those assumptions must be wrong.

So for the Fermi paradox, you might say, if life is everywhere, how can we not see anyone? Therefore, the assumption to revisit is that life is everywhere. And here, with the Red Sky Paradox, we might challenge the assumption that red dwarf stars are even capable of sustaining — and really specifically — complex life like us, observers. Maybe they have simple life, but something prohibits them from evolving all the way through to something that can do statistics, do astronomy, do geology — like learn about its planet and kind of essentially write the paper that I wrote about the Red Sky Paradox. That’s kind of the cogito ergo sum criterion I’m using as my conditional in this reasoning.

I have been making that suggestion to colleagues because the James Webb Space Telescope right now is heavily invested on red dwarfs. There’s a good reason for that. It’s kind of all they can do. Unfortunately, it just doesn’t have the capability, the technology, really to do anything with Sun-like stars. But red dwarfs, it’s game on. And I’m just saying, look, there might be reasons why it won’t turn up anything.

Henry Shevlin: And is the specific suggestion, I think I’ve heard, that basically in the early years of the early formation of red dwarf stars, they might be especially turbulent in a way that sort of scorches any planets in their vicinity and strips away their atmospheres? Is this one of the empirical predictions that we can make on the basis of the Red Sky Paradox?

David Kipping: I would say it’s more a consistency than a prediction. I try to be very careful. I love very broad agnostic reasoning as much as possible. In this case with the Red Sky Paradox, I don’t have to invoke any mechanism specifically. There is probably a mechanism, surely there is a mechanism — unless we are really a one in 1600 outlier. That’s possible as well, and I concede that it is possible, that we are just a very unusual example.

But if that’s not true — for a typical example — then there is some mechanism which bars the evolution of observers like ourselves. And in the paper, I point out there are numerous mechanisms people have suggested, including the fact that these stars have large coronal mass ejections coming off them, which can strip planets of their atmospheres. They have a prolonged, what we’d call adolescence for a star. Our Sun went from being born to being a main sequence star in the space of about a hundred million years, even less than that, tens of millions of years. Whereas red dwarfs take a billion years sometimes to settle down. And during that adolescence phase, they’re violent, and they can actually remove all the water off their neighboring planets.

We think it’s that, that’s when water gets delivered. Our water was probably delivered by comets during the late heavy bombardment and the other bombardments that were occurring before that. And so if during that time you’re delivering water through comets, the comets get depleted, but the star is so active it’s stripping all the water off them, then you’re kind of net zero — like you don’t end up with any water at end of the day. And then, when all said and done, you’ve just got dry planets around a normal star, but it’s too late. There’s no more water left to deliver to the planet anymore. So that’s been suggested as well.

Then there’s also the questions about photosynthesis. Is photosynthesis possible if the star is much redder than our own star? Because obviously plants on Earth use blue light as well as red light. If you take away all the blue light, how will they do? We don’t know. It’s kind of unclear. We don’t really have too many examples of life on Earth which thrives under those conditions. And then there’s tidal locking — these planets have probably one side of the planet facing the star.

So there’s many sensible concerns. But what I’m trying to do is avoid saying it’s this, it must be this one. Because that’s really for the astrophysicists studying the geology of those objects to figure out. I’m just saying there probably is something, and go after it.

Dan Williams: I’m not sure entirely how to frame this question, David, but someone might respond that there’s just something a little bit weird or surprising that you could draw seemingly substantive inferences from such a slim evidential basis. The starting observation here is just we exist where we do. And then there’s this interesting probabilistic reasoning. And then that’s leading you potentially to draw inferences about where life might potentially evolve in the universe. I suppose this is just an objection from the perspective of, isn’t there something a little bit weird about this entire style of reasoning?

David Kipping: It’s definitely weird. Yeah, it’s very weird. I always think Nick Bostrom is really like the father of all this kind of thinking in the modern era. And he often concedes that point, that it is very strange. We don’t really have a complete theory of anthropic reasoning. It’s sort of a work in progress, to some extent. In the same way, we don’t really understand how AI works. We don’t understand really the full nature of the universe. They are works in progress.

And yet it also seems logically, you can pose these logical questions that seem irrefutable or compelling. So like I mentioned the Weinberg example: it is really hard to imagine how you could possibly have the cosmological constant be a thousand times what it is, because Weinberg’s right — you just wouldn’t have galaxies. So how could you possibly have us in that situation?

The fine-tuning argument for the multiverse is the other popular use of it in modern science, I would say. They often point out, why is it that the gravitational constant and the fine-structure constant and the speed of light, all these things are just the way they are? There’s a simple anthropic reason for it. You don’t have to accept it, but you can certainly make this argument that if they were anything else, you wouldn’t be here to talk about it. So you can’t really change the mass of the electron by a factor of ten and get away with it. There’s going to be repercussions to chemistry. If you made the speed of light ten times slower than it really is, then relativistic effects happen in sort of everyday cases, especially for chemistry — that impinges the ability of electron shells to be stable. So you start to really ruin the CNO cycle inside stars and stuff. You start to ruin a lot of interesting nuclear physics and chemistry. So you can see, I think that’s the most common case.

There’s also a fun case — I think this is true, but it’s kind of a bit of an urban legend — that during World War II, there’s this thing called the German tank problem, if you’ve heard of that. The Allies would apparently — maybe you know better than I do whether it’s true or not — would see the numbers imprinted on German tanks. It would say one-five-five or something. So they would look at that number and say, okay, so they must have a border of like 300 tanks. Because if that’s a typical number, they’ve probably not got a million tanks, because otherwise it’d be very unusual that we had the 155th tank out of a million that were being produced. And they probably don’t have 155 tanks, because then we’d just be very lucky that we’d caught the very last tank that was manufactured. It’s probably of order three to 400. And so they used that to set the manufacturing constraints for the factories back in the UK — like, this is how many tanks you need to produce, because we think that’s how many the Germans have.

So yeah, there’s examples of this reasoning being used quite a bit. I think the one way it really troubles people is the doomsday argument. I think that’s kind of like the one that everyone gets — no, something doesn’t feel right about that when you apply it to that case.

Dan Williams: Could you walk through what the doomsday argument is, David?

The Doomsday Argument

David Kipping: Yeah, sure. So it’s been invented like three or four times, I think, by different people at this point. It essentially says that if we are a medium example of ranked humans that ever live — so you go from, I mean, this is where it gets a little bit, I always think, a bit ill-defined — like, you have somehow a first human who lived, I don’t know, a million years ago or something, and then you go all the way up to today, and maybe you count up that it’s of order of sort of 100, 200 billion humans who’ve ever lived throughout human history.

So if you’re somewhere in the middle, then you’d expect there to be about another 200 billion humans to go before we call it a day. And of course, the birth rate is much higher — there’s much more people than there is today, more importantly. So the number of absolute people that are being born is much higher than it ever has been in history. And so that means there’s probably only like five or six generations left, or something, before you run out of these people. And so that’s kind of disturbing because it implies that there’s only like a hundred or a few hundred years to go before doomsday will happen.

So a lot of people think that’s really weird. How could you possibly take your rank position and make inferences about the extinction of humanity? When it’s framed like that, I think it feels really flimsy. But on the other hand, if you frame it slightly differently — you look at like the Foundation series or Star Wars or something like that, where they have these galactic-spanning empires — and you think how many individuals must be living in those societies. They’re all humans, right? They’re humans just living all over these planets in the Foundation series. You’d have trillions and trillions and trillions of trillions of people. And the chance, if you were born as a random soul at a random time, that you would be on the progenitor planet, pre-empire phase, would be vanishingly small. So you might therefore make the argument that that doesn’t look like a likely future for us. It doesn’t seem likely that humanity will ever become a galactic or universal-spanning species, because how does that possibly make sense with us being so early in the story?

But there’s lots of ways to criticize it. One is that maybe humans change. Maybe the experience of a human in a thousand years from now is some kind of cyborg, or genetically modified version of us, or just natural evolution that — their experience is not the same as us. And so we can’t say that they’re a representative example. That’s kind of the key part of this assumption. You can draw a random member, but maybe the membership itself evolves in some subtle way.

And certainly that goes backwards in time, like, when does Homo erectus suddenly become human and suddenly not? It feels very artificial to draw a line. Do you include all animals that have ever lived by that metric? How does this work? So I think that’s where, when you start ranking people, it gets really flimsy. But I think this is more a criticism of the ranking aspect of the anthropic argument, and the anthropic reasoning itself. I think it’s more to do with the ranking — that it’s probably an ill-defined problem to try and rank and discretize people like this, because of the changes that happen to humans.

Henry Shevlin: So this is one thing that I get hopelessly confused about when I think about anthropic reasoning, which is sort of the reference class problem. How do you decide how to specify your sample? Because in the case of the Red Sky Paradox, you might say, well, I step outside and I see a yellow star, right? So of course it’s impossible that I could ever have been born around a red star. So you could condition the reference class on the type of observers living under yellow star atmospheres. Why doesn’t that diffuse the problem?

David Kipping: Well then you’re kind of like double conditioning. You’re almost like saying, what’s the probability of having water on your planet given that you have water on your planet? Well, it’s one. I mean, obviously it’s one, because it’s a double, it’s self-conditional, it’s a circular statement. Obviously you can certainly make such a statement, but it doesn’t teach you anything. So you can say, what’s the probability of having a yellow Sun given you have a yellow Sun? But it doesn’t move the needle in any way.

So you do have to make a stretch. And so that stretch here would be: what’s the probability of an observer seeing a yellow star under the assumption that observers are equally likely to be born around any type of star, or any main sequence star, to be a bit more specific? So that’s the tacit assumption. And it’s reasonable to question that assumption. That’s kind of what the Red Sky Paradox tries to do.

The reference class issue is a sticky one. And again, I think this leads to these questions of, do you use the self-sampling assumption or the self-indication assumption — SIA versus SSA? They can lead to different conclusions, especially for these toy problems like the sleeping beauty problem and things like this. And those are just unresolved. You can take the Sleeping Beauty problem and get two different answers depending on how you do the anthropic reasoning. So I think these are totally sound critiques of the model. But at the same time, we do have to concede that it has had some interesting successes along the way in its journey so far. So I give it some credence, but I’m also cautious about using it.

Henry Shevlin: One thing that’s troubled me about thinking about Red Sky-style paradoxes is it seems kind of implausible to me that we would be orbiting around — that we’d be sitting on a planet to begin with. Maybe I’ve just read too much Iain Banks, but it seems to me that the vast majority of habitable landscape across the future of the universe is going to be — for at least sentient, for sapient beings, let’s say the kind of beings you can do statistics — is going to be on orbitals or constructed habitats. So why do we look up — why are we on a natural planet to begin with, when you’d think that any sufficiently advanced civilization would be building artificial habitats? Is that also a puzzle? Should that lead us to think that people aren’t going to build habitats at scale, or the majority of sapient life that’s ever going to exist is going to be, for whatever reason, planet-bound rather than on orbital habitats?

David Kipping: Yeah, I mean, you’re kind of adding in this extra ingredient of what happens to super-advanced civilizations. Most people, if this is true, would probably be born off-world. Let’s just call it that. Whether it’s orbitals, or just another planet, or a moon, or something, they’d be born off-world — which obviously isn’t true. You were not born off-world, I was not born off-world. We don’t know anyone who was born off-world. So therefore it’s already an interesting constraint to some degree, that hasn’t happened.

A simple resolution to that is to say that just doesn’t happen. Species never get to a point where they do that. Or at least species that have a — and this is where it gets very philosophical — comparable sense of consciousness to us, or whatever that means. Because perhaps there is AI doing this, but we can’t be born as AI. Perhaps there are funguses which do this — technological fungi, that’s, you know, we can’t really imagine what they’d look like, but somehow they do that, and their experience of reality is so different to us that we should not be surprised that we were not born a fungus. It’s a meaningless question to even sort of frame it that way, because they’re colonies of single-celled organisms that just extend ad infinitum. So that’s where the reference class problem gets really sticky.

The one I’ve been thinking about the most recently — and it’s kind of a real classic one — is what’s called Hart’s Fact A. It’s considered the strongest constraint by many in SETI, the search for extraterrestrial intelligence. It’s that, again, we exist. And if you imagine extrapolating human technology, even a century, maybe even just a few decades into the future, we can imagine self-replicating, what we call, von Neumann probes. You could put an AI in a small chip, you could accelerate it — not to the speed of light, but even like 1% the speed of light would be more than enough to make this a real problem for astronomers. The size of the Milky Way is about 100,000 light-years across. So at 1% the speed of light, in 10 million years you could colonize the entire Milky Way. The galaxy is 10 billion years old. So that could have happened a thousand times over by now. And yet it clearly hasn’t.

So that’s startling because there are a hundred billion stars, a hundred billion opportunities. For someone, at some point, however unlikely it is — if it’s a one in a hundred billion event, then it should have happened by now. And we shouldn’t be here to even have this conversation. So that’s a really strong constraint, I think, that civilizations just don’t get to that point for whatever reason.

Maybe they don’t choose to do it ethically. It’s hard to believe there’s a universal ethics like that. And of course, these systems don’t have to be — it could just mutate. If you make a self-replicating probe, each generation will have errors. And so those errors will cause the behavior of the probes to change. You could very easily have these runaway situations. In a way, it’s like the most dangerous technology an alien could ever develop. And yet that seems to have not have happened. And that’s really interesting from an anthropic perspective, because it does imply that we’re probably as advanced as it gets.

Science, Philosophy, and Falsifiability

Dan Williams: One of the things you said there, David, was: this is when things start to get really philosophical. I’d be interested to hear your thoughts about how you view that relationship between science as it’s sort of conventionally or traditionally understood, and philosophy, and how you position yourself in terms of the relationship between the two.

David Kipping: I have no formal philosophy training, first thing to say. I always like to be candid about what I don’t know. I don’t have a philosophy background. I remember when I was actually thinking of doing undergraduate, Oxford at the time had a physics and philosophy degree — I don’t know if they still do. It was a double major, and I was really attracted by that. But everyone told me that Cambridge had the stronger physics program. So I thought, okay, that’s really my passion is physics, I’ll go for Cambridge.

I’ve always had an interest in philosophy, and I think obviously science naturally has a connection to it. Sean Carroll often complains about this, especially in quantum physics — there’s this kind of “shut up and calculate” view that a lot of us have adopted, where we don’t really, we’re not encouraged to think about the implications of our work. But sometimes the implications can shake you to your bones when you really think about what they mean.

And that’s what gets me excited. As a kid, what I was always drawn to is just asking, what else is out there? Am I part of some bigger continuum? What is the nature of humanity ultimately? I think natural philosophy obviously tries to address those questions in a related but slightly orthogonal direction. So I’ve really enjoyed at SETI meetings — there’s often the opportunity to talk to philosophers directly. There’s all sorts of backgrounds: anthropologists, social scientists, people working in media, obviously physicists, astronomers. So you get this really diverse group of academics, even theologians. I think theology has lots of interesting connections to looking for aliens, because God and aliens actually have lots of similarities. So it’s really fun at those meetings to have — it’s the only meetings I go to where you get that kind of broad interdisciplinary interaction. So that’s where I’m learning most of my things and having those great conversations.

Dan Williams: I once had dinner with Roger Penrose, and he said that the people he most enjoys talking to are philosophers of physics — actually, philosophers of physics at Oxford — rather than physicists, precisely because he thinks with many physicists there is this kind of “shut up and calculate” mentality. They’re not willing to engage with those really kind of big-picture, fundamental questions.

But I suppose another way of coming at the same question about the relationship between science and philosophy, and how you view that relationship, is: what’s the role of kind of ordinary empirical testing when it comes to addressing these really big-picture questions that you’re engaged in?

David Kipping: Maybe this isn’t directly answering your question, but one connection that comes to mind when I think about that is Popperianism, and the definition of the empirical process of the scientific method. We have this guideline from Karl Popper, which is, your theories have to be falsifiable. Otherwise it’s not really science. You’re doing something else. And a lot of us have adopted that for a long time. Not really thought about it too much, but we were taught at college and then just went off with it.

But suddenly a lot of science that’s happening right now challenges that Popperian view. I have colleagues like Grant Lewis, who’s a cosmologist, he works on fine-tuning, for instance, and string theorists often would be in this boat as well — where what they’re working on doesn’t make any testable predictions. Certainly not in a practical way. Maybe you could imagine in some extremely advanced civilization, we’d have to build particle colliders that could be galaxy-spanning wide or something, to test some of these theories. But typically they’re asking questions that are unfalsifiable.

And even questions that I’m interested in, like, does Mars have life on it? That’s, to some seminal degree, actually unfalsifiable. I can’t ever prove that Mars is sterile, because there’s always another rock to look under. There’s always another core drilling site you could dig under to see if there’s someone there. So you can’t ever disprove it. And I can’t disprove that UAPs are aliens. I can’t disprove that aliens are not inside your body right now and you’re just wearing human skin. You can go down this slippery slope kind of view where everything just becomes unprovable in science.

But I think bringing it a little bit back to cosmology, they’ve been saying — at least Grant has been telling me this, I’ve been thinking about it a lot — that it doesn’t really matter whether it is falsifiable. It’s whether it has use, is it useful? It’s kind of maybe a better way to think about these models. Certainly the multiverse, even though it’s not testable, it has explanatory capability through that anthropic argument we talked about before. It can explain why the constants of the universe are the way they are. And if you don’t have that, you just would have to accept it as brute fact, or hope for a miracle, which is to say that one day physicists will figure it out and there’ll be some reductionist view to explain where it comes from. But it’s also possible that will never happen. I think it’s quite plausible that will never happen. And so then you’re just sat with brute fact versus, at least this has explanatory capability.

It doesn’t prove the theory is correct. I don’t think you can do that. But you can say that it’s useful. And when you frame it that way — I think a lot of us would say quantum theory isn’t really true. It’s just useful. We don’t really know to what degree the universe truly is quantum. There might be some deeper theory, as Einstein suspected, that explains all of these random probabilities, and we’ve just yet to uncover what that deeper theory is. There’s some grand unified theory beneath it. So the model of the universe being quantum is an extremely useful model for calculations, but we shouldn’t necessarily assume that it’s a totally accurate description of how the world really is. So perhaps this falsification then might be challenged as being — well, let’s just find things which actually explain stuff, and we can use in our society to progress things.

AI in Science

Henry Shevlin: So I think probably these issues of philosophy and science and their relation are going to continue to percolate in the conversation. But I’d like to take us now to discussing AI a little bit, because there was an absolutely fantastic recent episode of the Cool Worlds podcast called “We Need to Talk About AI,” which seems to suggest that, at least for you, this was a real wake-up call. I think it was one meeting at the School of Advanced Studies in Princeton. Do you want to just give us a quick summary of what this meeting meant to you, and how it was maybe shaping your views on what AI is doing to the sciences?

David Kipping: Yeah, so this was a meeting, I think in February or January — it was a few months back now, near the start of the year. I think like many people, many scientists I know are using these AI tools. And I was certainly using them. I wasn’t using Claude at the time, but I was using ChatGPT a little bit, and Copilot, and things like this. I kind of assumed that the really smart people — because we all have a bit of imposter syndrome — don’t do that. The really good coders don’t need Copilot. They’ll just code up properly. They’ll do their reasoning without any help. And I was using it as a crutch because I was inferior to these other great scientists. And so it was just sort of helping me in that way.

And then what was startling was at this meeting, these people though, just have the highest respect for. Because the Institute of Advanced Studies, you know, it is like the pinnacle of where you can go intellectually amongst many other schools, but it is one of those very, very top tier places. I remember I walked down the corridor and saw Ed Witten. People say he’s got the highest IQ on Earth — they say that about Ed Witten, right? And so you’ve got people like that saying they’re all using AI tools for not just coding. And these people were like hardcore coders. They were writing these — Enzo and Gadget — these like astrophysical simulations of galaxies and hydrodynamical fluids and stars and things like this. Really, really complicated codes. Legacy codes that have been handed down sometimes over advisor to student to student to student generations of people. And they were using it.

So there was a concession that it has coding supremacy. That language was used — that it already has coding supremacy, and we have to admit that and use it. It doesn’t make any sense to pretend it doesn’t. And second, that it possibly has mathematical supremacy. There was — it was less certain — but there was a sense that it was already pretty close to being as good as what we can do mathematically, even in some cases superior. And that was really wild to hear. To me, it just sort of made me think, I’m not being like the idiot in the room by using this. Everyone’s using this at this point. And if anything, they’re trying to accelerate the adoption of these tools, not resist it. There was no way back, sort of view, about it.

Henry Shevlin: And of course, David, you’ve been using AI in the broad sense basically for your entire career, I think. Have you seen significant evolution in the way these tools have evolved? Was there one moment, perhaps it was this meeting at the Institute of Advanced Study, where things suddenly kicked into a different gear? Or have the tools been steadily improving since you started in the field?

David Kipping: Yeah, certainly in my own career, I was more on the development side of some of these tools for a while, but not at a serious level. We wrote a couple of papers where we developed our own deep neural networks — just simple feed-forward, back-propagation trained models for bespoke problems in astrophysics. In particular, we were interested in predicting if you take a solar system, can you predict whether it has additional planets in it? Questions like that. And then where would those planets live? So we could take this sample of all of these known planets and make successful predictions for the systems.

I’d written my own DNNs like that. It was mostly — I mostly did it, I think, because I was just interested in how they work. The best way to figure out how something works is just to find a pet project and code it up. So I was more on that development side. That was sort of 2010, 2011. And then in the years that followed, I started to back off it, because lots of astronomers were doing AI — and still are — but what I was seeing was that it wasn’t like a hobby project anymore. You couldn’t dip into it and mess around and write an impactful paper, and then go away and do Bayesian statistics and all the other stuff. It was becoming a full-time job, because the literature was just exploding. To keep up with it was like you would have to spend all your time just reading the archive and playing around with various AI tools to keep up with that.

And I just consciously decided I didn’t want to do that, because AI is not my passion. Science is my passion. So I kind of left it to the wayside. I’ve said to several students recently over those years — they were like, “I saw you did these AI projects. Can I do one with you? I’m really interested in AI.” And I’m like, I’m not doing anything else with AI at this point. So I kind of went stagnant on it.

And then most recently, I’ve now become, I’d say, like a power user of it. I don’t have any false narrative in my mind that I’m going to develop the next LLM for exoplanets, or for anything. That’s not my interest. There’s no point. I can’t possibly write an LLM anywhere near as good as what OpenAI can do, or Anthropic can do. So I may as well just use the tools, and think about how to use them as effectively as possible in my field. I think that’s the transition that I’m seeing a lot of people moving to — that the billions and billions of dollars of investment these companies have make it just a complete waste of time for astronomers, especially, who aren’t even software engineers, to possibly try and compete with that. We may as well just try and use them in a way that advances our field.

Dan Williams: So in terms of the use of AI in science now, as you said, David, there are some people, including some of the smartest people on the planet, who are using AI aggressively. There are some people both inside academia and outside of it who are aggressively against the use of AI. How are you thinking about that in terms of — are you really excited about where this is going? Are you worried about it? Do you understand some of the worries people have about the use of AI in science?

David Kipping: Yeah, for sure. It is, in some ways, it has analogies to what’s happened before. One concern might be the ethical concerns of how much power, especially for climate change — how much power and how much water these data centers use. Even potentially, building space data centers would also be a form of further contamination and pollution to our natural environment. So I think you could understand why someone might say, “I’m trying to be carbon neutral, so I just don’t want to use these things.” But that debate’s already — that’s not a new debate, because astronomers have been using high-performance computers for generations already, since probably the ‘40s or ‘50s. As soon as computers were accessible to scientists, astronomers were using them to do big calculations.

I remember there was a really fun paper, like about 10 years ago, that made a lot of controversy. It was saying that all astronomers who code in Python are bad for the Earth, because Python is so computationally inefficient that you are basically emitting 10 times more CO2 than you need to if you just coded in C instead. It was like really trying to shame astronomers who coded in Python — of course, basically all astronomers these days code in Python. So a lot of people really didn’t like that paper. But it was a fair point, like if you really care about your carbon footprint, then that’s a big factor — these data centers, what they produce.

So that’s not that new. Different people will just arrive at different comfort levels as to where they think these tools are applicable. There’s also this kind of oligarchic element to it as well, like these companies and the extreme wealth and the wealth inequality in our society, the future of work, the future of labor — all get tied up into that. So it intersects so many things.

I think it’s interesting that AI has become such a political topic. I think it didn’t used to be that way. It used to just be like a tool, and you had an opinion about the tool, but now it’s like very politicized. And even, I’ve noticed that some students who identify as very liberal will not use AI tools. And maybe students who are more right-leaning or centrist will not really care as much about that. They’ll be like, “well, whatever, it’s just the way of the world. Let’s just be pragmatic about it.” Even saying you’ve used AI can certainly trigger a political reaction to your work, if you say that. So that’s, I mean, this is all kind of new. That was very on the margins when previous work I found with data centers and high-performance computing. But now it’s becoming much more present. So that’s interesting.

I’ve just been thinking personally — I think the question I’ve been asking myself is, I’m on sabbatical right now, so I don’t have to deal with it, but: would I hire a student who refused to use AI? I talked about that, I think, in that podcast episode, and I’m still thinking about that. I think I probably wouldn’t, in the same way that I probably wouldn’t hire a student who refused to use the internet. It would be such a disadvantage to them. If they said, “I’m only going to use a typewriter, I’m not going to use a computer,” I’d be like, okay, that’s fine, but you’re really tying two hands behind your back here. If you want to get a job, and you want to have an impact for PhD, and we want to get some work done together — you need to be using these tools. It’s weird not to use them. So that’s a difficult conversation to have with yourself and with the student, but it’s certainly something I’m thinking about.

Henry Shevlin: So there’s a related worry about the impact of AI on sciences that I think has come up a few times on the podcast, most recently with Chris Lintott — about whether AI might strip science of a lot of its human value. If we’re relying on AI systems to produce the next generation of theories that may be to some extent inscrutable to humans, that this will sort of destroy the most successful project in human history, namely humans doing science. And I guess the counter-argument to that is that the reason that we fund science at scale, the reason we build particle colliders and expensive space telescopes, is because we care about results. So fine if people want to be hobbyist scientists to experience the joy of science. But should the taxpayer be funding your own epistemic discovery and aesthetic enjoyment? Or should the taxpayer be concerned about results? So I’m curious where you land between those two positions.

David Kipping: Yeah, I think I was a lot more concerned about this a few years ago. And weirdly, I’ve actually gone the other way a little bit. A few months ago, I was right with you. I was really worried about — what’s the point? I don’t want to live in a world of magic. I want to — the point I became a scientist is because I want to understand how things really work. It’s understanding. And I don’t want a model just to spit out a result, have no idea where it comes from or what it does, and just trust it. That’s not comfortable to me.

But having used these models a lot over the last few months, I’ve become — A, you get a bit acclimatized to using them, but B, you start to understand the limitations, at least of the current versions of what it’s doing. And it’s certainly not at the stage where it’s able to pump out a paper. It’s just not there at all, in my opinion.

There was a colleague of mine who spoke to me about this recently, where she had a PhD student who wrote a really nice first draft of a paper, a really great astronomy paper. They submitted it for review, and they got the referee report back. And then the student came to her a few days later and said, “I’ve finished the second revision already.” That was quick — just two days. That was fast. And she looked at it, and it was just complete nonsense. The paper was twice as long. All the figures were ruined. It was overly verbose. The messaging had just completely been lost. She said to him, “did you put this into ChatGPT?” And he was like, “no, no, no.” But then it turned out, of course, she did. Eventually he confessed that that’s what he had done. So they had to just totally scrap that revision and go back and do it the old-fashioned way.

I think that’s just a good example of how — I mean, it kind of touches on also expertise, like — I don’t think a senior person at my level would do that. But I think students and interns could be tempted to do this, where you just do that, copy and paste the whole damn project into ChatGPT and say, “do it.” That’s really dangerous in my experience. And it’s not the correct way to use them. You need to figure out a plan in your head a little bit, or even interact with it to develop a plan. But it has to be like a conversation. And then you need to go piecemeal — you take little bites of it. You ask it to pursue that next thing. You test it. You compare it to other codes you know that do the same thing.

In a way, that’s not that different from what scientists have always done. To go back to the example of using large-scale simulations of the universe — if you’re a PhD student who is trying to simulate, I don’t know, supernova feedback around supermassive black holes, or something, the star formation regions around those areas — you might be handed over surely a giant piece of code, hundreds of thousands of lines of code that have been handed down over like 10 years of people developing it, with huge teams. You would not be expected to understand every line of code in that. You would be expected to use it, and to understand sort of broadly what it’s doing, and to ask skeptical questions. So if you got an answer that said there was negative star formation, you would look at that result and say, hmm, that doesn’t make sense. Let me work through the problem and see where it’s going wrong.

It’s that kind of sanity check that I think physicists, especially, have always learned to do — those back-of-the-envelope calculations. Yes, you have some sophisticated computer code that spits out impressive answers as a black box, but the skill of being able to check things with your brain and ask those reasoning questions is absolutely vital. And almost every time I use these AI models to do something, it messes up the first time over, and I catch it out, because I’ve done that back-of-the-envelope calculation. I’ve said, well, actually, let’s take the asymptotic limit of this in this limit, or this degree, and you can see it fall over. And it’s like, “oh yeah, you’re right.” And then it will go back and fix it. But that’s that vital skill that I think we’ve always needed.

So I don’t know — I don’t know how things are going to improve. Maybe eventually it’ll be able to do all of that itself, and just completely take over. But certainly, as impressive as Opus 4.7 is, and these are the models — they’re nowhere near that level yet, in my opinion, of being able to run away and do science.

Dan Williams: So the obvious argument, you suggested, David, for scientists making as much use of AI as possible is that it’s just going to help them with the work of science and advancing the frontier of knowledge. That’s kind of the social responsibility of scientists. Can you foresee any ways in which actually, even though it might seem like it’s making us more productive, it might have some negative consequences for that core scientific project of creating and advancing knowledge?

David Kipping: Yeah, certainly there’s spamming, which can happen. You can have — and that’s been happening in some journals. I don’t think astronomy journals have suffered from this too much yet, but there are certainly examples of people doing what that student did, which is what you shouldn’t do — which is just to prompt an entire research project and not really look at it too closely, and just submit it to a journal. The journals themselves may start using AI to do the refereeing — again, in which case you could just end up with an enormous amount of, what would, AI slop literally in these journals.

What I worry about — I mean, it’s true with image generation as well, and other things — is just that kind of recursive loop then starts to close. You start to have scientific agents that are trained on junk. Because if we get to a point where there’s enough junk science out there, then what it’s learning is junk, and so the true scientific innovations get lost in the noise. So that would be really worrying.

I do think that human referees are a vital part of making sure this doesn’t happen, which is an interesting problem because human referees are in very short supply. It’s very hard for editors to find human referees these days. But yeah, in the same way that that’s happening with music, and it’s happening with image generation, and it’s happening already with video — I think it is a worry that you start to train on fake data.

I know that — I was listening to the NVIDIA CEO, I forget his name, he was on Lex Fridman recently —

Henry Shevlin: Jensen Huang.

David Kipping: Yeah, sorry. He was talking about how they’re very comfortable with using simulated data and augmented data. I don’t really know how that would translate to science. It would make me nervous to generate fake scientific papers and then train on them to create an AI researcher. I’d have to think about that and learn more about what they had in mind there. I don’t think he was thinking about research particularly in that case, but it would have to — you’d have to solve that problem, because you probably wouldn’t have enough volume for, in terms of research papers, really to create credible agents, at least with the training tools they’re currently using.

AGI Timelines and the Future of Science

Henry Shevlin: So you mentioned, and I completely relate, that current AI agents — although they’re very useful as tools, they can’t take over large-scale project management single-handedly, particularly in the sciences, or in my own field. I find AI tools very useful when doing, for example, research for philosophy and cognitive science papers, but I wouldn’t trust writing a paper to one of these things anytime soon. But at the same time, the timelines that serious researchers are talking about — they talk about five, 10 years away from AGI, from real transformative super-intelligence. And I’m just curious whether you are skeptical of some of those timelines, or whether you see real transformative AI in our near future.

This actually really comes across, I think sometimes in the show, when you’re talking in the podcast — when you’re talking about, you know, various new telescopes that are scheduled to go up in the 2040s. And part of me just thinks, come on, by that point either all of the major predictions from leading labs about the destination of AI, AGI, will be falsified, or these telescopes will be — maybe not redundant — but our sights will be set much higher. We’ll be building our first Dyson swarms by 2045. So I’m curious, are you a skeptic about some of these more ambitious goals for AI in the next decade or two?

David Kipping: I’m certainly a skeptic of having Dyson swarms, I’d say, by 2045. That would surprise me a lot if that was true. Because I think there’s a big difference between software and hardware — actually to physically build stuff. Even what’s slowing down a lot of this development with AI is they can’t build data centers fast enough, nor the power to supply them fast enough. Energy is really becoming the bottleneck for them, not the software development.

I always try to be very agnostic about everything scientifically, especially about predictions of the future. And it’s totally plausible that there’s a ceiling — that there’s a ceiling to how good these models can get. Usually that’s true of most things. Most things are S-curves. There’s hardly anything in the universe that’s truly exponential, except for probably the expansion of the universe. That’s the only thing that’s exponential. Everything else is an S-curve in nature. So it would be weird if it didn’t saturate at some point. And I’m not exactly sure what that bottleneck could be, but it could just be a fundamental limitation of large language models themselves.

The actual way we think — although language is an integral part of how we think, and obviously you guys know a lot more about this than I do as cognitive scientists — but it feels to me that there’s thoughts I can have that don’t involve language. I can imagine a ball rolling down a hill, or a spaceship taking off, and there’s no words in my head. It’s almost like a little physics simulation that’s playing in my brain. And I don’t know if the way these LLMs work will guarantee that it can do all the cognitive things I can do. I just don’t know. I’d be interested to hear what you think about that.

Henry Shevlin: Well, just to push back slightly, of course LLMs are one of many different games in town at the moment. You’ve got things like AlphaFold, GNoME, doing sort of basic material science research. I would have shared some of those doubts a few years ago, but seeing, for example, the amazing work being done by frontier AI in even LLMs in things like mathematics — we’ve now had multiple Erdős problems being solved with AI playing an absolutely central, defining role. So I’ve been surprised at how well these models that seemingly just start out as linguistic predictors can actually contribute to frontier mathematics — LLMs and frontier material science or biology when talking about non-LLM AI systems. So I see the current wave of AI, although LLMs get all the headlines at the moment — we’re investing in multiple different pipelines in parallel.

David Kipping: Hmm. Yeah, that’s fair. I think the best case of agnosticism I can give you that I’ve used in my own work that bears on this would be the simulation argument, actually, which kind of leaps back to that anthropic point. You’ve probably heard Musk say this and others — that he’s stated very confidently that the odds that we don’t live in a simulation are like a billion to one. Like, we almost certainly are simulated, by this reasoning that, you know, if a universe can make a simulated universe, and that one can make a simulated universe, and so on and so on, then you’d end up with far more simulated universes than real ones.

But I point out in a paper a few years ago, very simple argument, that we don’t know that we’ll ever have the ability to make those simulations of that fidelity. Maybe there’s some bottleneck to our own ability. And what Musk was doing was taking one of the trifecta — the trilemma — that Nick Bostrom took, and just saying it was the last one was true: that essentially we would indeed go on to make these simulations. But there’s the other two parts of the trilemma — A, that we never develop the capability, or B, that we never choose to do it. So if you just have a more soft prior, more agnostic prior, you’d say, maybe there’s a 50% chance, or something, that we will develop that technology. There’s also a chance that we won’t develop that technology.

I just try to remain agnostic like that with AI, because if you just extrapolate all technologies ad infinitum, then you would certainly conclude with simulated. And historically, that’s been precarious. Percival Lowell took canals being built across America and said, that’s what advanced civilizations will do. They’ll just be covered in canals. And it seems silly to us — like, we think, why is that so silly? Why would a civilization cover their planet in canals? But to him, it made perfect sense as an extrapolation. Scientists today talk about tiling planets with solar panels, because that would be a natural extrapolation of renewable energy. And similarly, I wonder if in a few generations, the idea of extrapolating the capability of AI without any bound would look foolhardy. So I just try to remain totally agnostic about it. It is possible — I’m not saying it won’t happen — I just try to remain agnostic. I don’t know how far these things can go. I don’t think anyone really knows.

Dan Williams: Yeah, I agree with that. I don’t think anyone really knows. I’m also extremely uncertain about the timelines here. Just to double-click on one thing — state-of-the-art LLMs these days aren’t only trained on linguistic input; there’s sort of multimodal inputs as well. Although I also share the potential skepticism about whether this particular kind of architecture will scale to AGI and super-intelligence and so on.

But David, suppose we fast-forward five years, 10 years, and we do have AGI, in the sense of AIs that can fully substitute for the kinds of stuff that we do — for all kind of economically valuable, scientifically valuable human labor. How would that cause you to update your views about these other big-picture questions you’ve looked at? You mentioned the simulation argument. Earlier on, we touched on the Fermi paradox. So I totally take the point — there’s huge uncertainty. Suppose that resolves in 2035 and we do have the real deal, super-intelligent AI. How would that then shift your beliefs about these other topics?

David Kipping: Yeah, it’d be a big shift, I think. It’d influence all sorts of aspects of this conversation. One thing we see already with these AI models is how energy hungry they are. And if you extrapolate that, then surely the only purpose of these computing data centers is to compute as much as possible, as fast as possible. And so that implies that you’re going to need vast amounts of energy.

One interesting consequence that I’ve been thinking about just recently is, with these orbital data centers that billionaires are getting very excited about — that would produce quite a signature. We should probably see that in James Webb data. We could probably already put limits on the existence of essentially artificial rings of thermally hot — because they’d be emitting a lot of infrared because they’re warm — geosynchronous orbits, most likely, to capture as much solar energy as possible. So that puts them orthogonal to the plane at which these planets transit. So that maximizes their detectability. So I think we should see that. That gives you lots of ideas about what might be possible to do with asking these questions about other life.

But if we make that breakthrough, I think the biggest point is it seems to imply that we are alone. Because if we can do it, surely someone else could have done that. And it really does exacerbate that point we talked about earlier with Hart’s Fact A — that we seem to live in a totally natural universe. Everything about the universe we see — stars, galaxies, clouds of plasma — everything is consistent with nature. There’s no hint anywhere of anything artificial, no engineering, nothing in the whole universe as far as we can say is true. That is weird.

If we can invent these machines which have this exponential capability to just basically almost do magic — just do whatever they want, Dyson spheres everywhere, colonize wherever they want, faster-than-light spaceships, whatever it is — it just massively exacerbates the Fermi paradox, to the point where you’d probably conclude this is it. That would be my natural reaction. It would make me even more pessimistic, I think, about the probabilities of civilized, intelligent life in the universe.

Henry Shevlin: I mean, there’s a fun idea here that if we do develop AGI, then this should massively raise our prior on us being a simulation, which could also — and the simulation theory is sometimes offered as an explanation of the Fermi paradox itself. The kind of pop version of this is the kind of “draw distance” argument that you see from video games. If you’re in a video game and you look at the mountains in the distance, they’re not fully rendered. They’re just like a skybox, right? So in some sense, you might say, well, the reason we haven’t found a universe paved with technosignatures is precisely because we’re in a simulation. There’s no point simulating — if you’re doing an ancestor simulation of life on Earth, then you just need the minimal amount of background information in the galaxy.

David Kipping: Yeah, I agree. It comes back to this idea of, what is science? Because I think simulation theory has explanatory capability like that. It naturally explains why there’d be no one else out there. And it also kind of explains why we live when we live, right? Because we would live, basically, in the most interesting time, which we seem to indeed live in — the most interesting time of this step-function transformation, where you might be interested in seeing how does that play out? What does that look like? Let’s simulate it. Let’s see how it looks. So it has a lot of explanatory capability.

But the simulation argument definitely fails the Popperian definition in most versions. Because any errors — you know, people talk about looking for glitches in the matrix — but any errors, you could always just rewind the simulation a little bit, fix the error, and then start back from before that error crept in. You could always just have reverse tracking. Go back to the last save game before you jumped off the cliff, right? Is what you could always do. So in that sense, I don’t think it’s testable. I don’t really know what to do with it as a scientific idea, except as an interesting philosophical idea. I think it would always be unprovable. It would always just be something we suspect — and maybe a lot of us suspect it — but we’d never be able to prove it.

But the idea of an AGI that can do everything I can do, just to reverse track a little bit, would be — it just changes everything, right? Because then what would I do with my time? I don’t even know. What would I — how would I spend my days?

Henry Shevlin: Well, hopefully producing — continuing to produce the podcast for a start.

David Kipping: But you wouldn’t need me to produce the podcast, right? It would do that as well. There’s no function to that, because you could probably think you’re watching me, but it’s just an emulation of me. You’d just say, “create fake Davids that make podcast episodes every two seconds.”

Henry Shevlin: But you see, this is an interesting argument about employment in the post-AGI era — that relational goods, or goods where the humanness is sort of the point, will become the most valuable area of the economy. A simple example here is the famous string quartet argument: I can play a beautiful recording of the greatest string quartet in the world, but people still hire humans to do it for them, because the humanness is sort of the point. I think things like entertainment might be an area where there’s a known person with their own brand and their own reputation. Maybe this is exactly the kind of area where humans will still be working, even if it’s AIs behind the scenes doing a lot of the science, doing the economically valuable activity in industry.

David Kipping: Yeah, but I do think a podcast is a digital product. That’s the deliverable. I actually physically upload a file to YouTube, or to Podbean, or whatever. That’s the final deliverable. So if you could produce that convincingly with an AI model, it’d be far easier for me to do that than to actually sit down for two hours, and I’d probably enjoy it less. I’m sure I would. So maybe we’d all just revert to actually physically meeting again, and talking in public lectures and things like that. Maybe that would be all that would be left.

But even then, it’s hard to imagine. If I tried to imagine giving a public lecture in 20 years time, after AGI, I’d have no idea what’s going on with AGI. Because AGI would be so far ahead of me. All I could talk about would be classical learning. I wouldn’t be able to tell you anything about how this latest discovery works, because it would probably be beyond my comprehension. And so that’s where I just lose excitement. I can’t even really imagine staying a scientist, because it just would feel purposeless. If I don’t understand what’s happening, if I’m not a participant — I mean, David Hogg wrote a wonderful piece about this. Maybe you saw it on arXiv: that ultimately we do science because we want to participate in science, not because we just want to have these answers delivered to us. That’s a byproduct of it. But ultimately, we’re curious creatures. That’s a fundamental part of human nature, is to want to understand how things work. And if we lose that, we just become spectators. I think that’s really tragic. So I fear that future. I would not really want to live in that world.

That’s why you’ve had — it hasn’t happened so much recently — but you had a few years ago people, like Max Tegmark, having these calls for pauses on AI development and things like this. I’m sure in part that’s fueled by asking these questions about who are we in that world?

Henry Shevlin: Have you seen this lovely Ted Chiang short story — flash fiction in Nature — called “Catching Crumbs from the Table,” from about 20 years ago? Where he talks about this era of post-human science, and he imagines that you have this new industry of machine hermeneutics, where humans try and figure out — try and explain in very dumbed-down terms — what it is the machines are coming up with. So that’s one vision of what the next generation of science could be: us consulting the sacred texts almost. They produce these amazing advances and we try to win out the sense and the logic in them.

David Kipping: Yeah, but even that, you could imagine AI doing that. I think that’s the problem. There’s really nothing — because the whole point of AI is it can do everything we can do. So then there’s nothing left for us to do. You can retreat and retreat. And especially if you get to the point where robotics can obviously do all the manual labor, and even eventually the emotional labor, and therapy, and talking to people. People talk about AI girlfriends already all the time, but God knows what’s going to happen once we have robotic girlfriends like that. It’s just going to be the end of the birth rate. That explains the doomsday argument, I think, right there.

It’s a terrifying future if all those predictions come true. But I just — something doesn’t feel right about it to me. There’s just like a spider sense, an intuition, that these models will never be able to replace everything that we can do. I think our role will evolve as scientists, as managers, in terms of where we interact with each other, as communicators in the media space. I’m sure all of that will evolve as it always has done. I am skeptical it will totally be displaced, because I think a lot of people don’t want that. There’s no — most people don’t desire to have no function in this world. Most of us desire to have a role. If humans don’t want it, I don’t think it will happen.

Dan Williams: I think it’s also important to distinguish the question of whether AI could replace human beings, from whether AI could replace human beings using AI and augmenting our capabilities and extending our capabilities with the use of AI. I think we are, as the philosopher Andy Clark puts it, kind of natural-born cyborgs. We’ve always extended our capabilities with the use of technology. I think even once we’ve reached really advanced AI, the period that will follow that will not just be us becoming, sort of, 19th-century aristocrats playing frivolous status games. I think there’ll be this long period where we’re augmented with this technology, rather than replaced by it.

The Fermi Paradox and Being Alone

Dan Williams: I have a question, just to go back to the Fermi paradox. It seems like many people have the intuition that if it is in fact the case that we are the only animals that creates super-intelligent AI, there’s something kind of surprising about that, just given the scale of the universe. As someone as an outsider to this whole literature, it strikes me there’s always something that seems sort of teleological in the way that that assumption gets set up — as if there’s some tendency in the universe towards intelligence and then technology and civilization. If we were the only animals in the universe that ever produced the music of the Beatles, I wouldn’t find that a priori very surprising. It’s purely contingent that that specific chain of events happened. Similarly, when it comes to the fact that we’ve got the cognitive capabilities and the institutions that enable us to build things — I don’t think there’s any tendency in the universe that’s pushed anything in that direction. I think it happened through lots of chance events, and through an evolutionary process that is not in any way kind of teleological. So what’s supposed to be sort of surprising? What’s giving the Fermi paradox that paradoxical character, according to many people?

David Kipping: Yeah, I mean, certainly when you look at human history, we were more or less biologically the same as we are today for the past 200,000, 300,000 years. And yet we did not have agriculture, the Neolithic revolution didn’t start until about 12,000, 11,000 years ago. So we were quite happy for 200,000 years to be hunter-gatherers. We weren’t compelled to develop cities and farm. They thought — I don’t know what they thought — but apparently they were quite satisfied with that way of life.

So it’s certainly not obvious that you could take even humans and put them in a different planet and rewind the clock and get the same outcome again. Maybe this is a very unusual outcome of what happens even in the human experiment, let alone other advanced intelligent beings out there. And of course, intelligence is so diverse, because there’s lots of intelligent creatures on our own planet, that it’s hard to imagine them developing a civilizational, technological civilization — like a dolphin, or something. Obviously, it doesn’t have the fingers and thumbs to really build anything like that, despite possibly having greater intelligence. We’re not really sure.

So there’s certainly no guarantee. But I think the argument might be like monkeys on a typewriter — that if you give enough rolls of the dice, you probably will at some point form a roaming AI. And if we do it, then it proves that that is the case, that it can at least happen in some instances. The question then becomes, what are your priors? Like how often do you think that happens? In a hundred billion stars, do you think that’s a probable outcome or not?

I did a calculation last week — I might publish it on Galaxies. Again, I used [AI] actually to help me with the math, to be honest, to go through it. But I just kind of asked: imagine each galaxy gets a chance of turning AI — I call it berserker, like just getting infected. There’s some spontaneous spawn rate at which a galaxy can convert from essentially just a vanilla galaxy into a berserker galaxy. And berserkers send out a signal at the speed of light, which — every galaxy they come into contact with, they infect. So it’s almost like an infection-type problem. But on top of that, you’ve got cosmological expansion — the universe is physically expanding on top of this as well. So that was the calculation I did.

It turns out that in order to get 50% of galaxies infected in the universe, the spawn rate is one in six billion galaxies. So if just one in six billion galaxies, over the entire history of the universe to date, ever spawns an AI, half of all galaxies would be gone by now. That’s even more so, because that’s one in six billion galaxies. Each of those galaxies contains 10^11 stars. So this is where things get — the numbers get really big and you start to run into real problems. Now you’re talking about an event that’s a one in a trillion level less than that event of happening. That’s where it just starts getting a bit uncomfortable. Maybe that’s where you start to think simulation thoughts, because you think, how does this make sense? How can there be just absolutely no one else out there? Because if we’re only a few decades away from doing this, what gives?

Henry Shevlin: So I think that’s a fascinating point. To pick up on something you said, Dan, and also something you said, David, about rewinding the clock. Stephen Jay Gould had this famous radical contingency thesis: if you rewound the clock of evolution on Earth, to what extent would we see the same kinds of animals and forms emerging? And as someone who dabbles in the philosophy of biology world, my sense is that there’s been a slight move towards thinking there’s perhaps less contingency than we thought. We see many instances of convergent evolution, convergent intelligence across, for example, eusocial insects and humans and cephalopods and cetaceans. And even at sort of earlier stages in development, primary endosymbiosis occurred at least twice, we think; multicellularity something like 20 or 30 times independently.

To relate this to the point you just made, David, when you think about the various possible locations of a Great Filter — there aren’t as many good candidates, perhaps, I think, as there used to be. Apart from perhaps the origin of life itself, maybe the emergence of something like eukaryotic life. But you really need those numbers in the Drake equation to get down, you know, to reach the trillion-to-one levels. So I’m curious — I know obviously you’re writing a book about how we might be alone in the universe — and I’m just curious where you think the filter is, or what the best candidates for the filter are.

David Kipping: Yeah, the origin of life, I thought, would be the obvious place to put it as well for a long time. Just because — certainly if you ask, what is the chance of making a protein by random chance? Take some amino acids — there’s 20 amino acids in a protein, 20 different types. A typical protein is like 80 to 100 amino acids in length. So therefore, the number of combinations ends up being, I think it was 10^180, possible ways of arranging those amino acids, and only one would be a protein. So it just seems like — we’ve never done that. No one, as far as I’m aware, has ever taken amino acids, shaken them up in a lab, and got a protein out of it. It’s such an improbable arrangement to form even a protein. So you can certainly make the argument [for that as a filter]. But maybe there’s — I think the counter argument was always, well, maybe there’s something we’ve yet to discover. There’s some autocatalytic process that’s making those that we have yet to find.

So I think the strongest piece of data we had in my mind for life elsewhere would be the occurrence rate of abiogenesis — was how early life started on Earth. As you say, similar to the evolutionary convergence aspects, that has been revised significantly over the last 10, 20 years as well. In fact, there was a paper in Nature a couple of years ago — maybe it was last year — by Moody Adow that looked at the genetics of LUCA, the last universal common ancestor, and estimated that it lived 4.2 billion years ago. Which is almost immediately, because the Earth had oceans — formed about 4.4 billion years ago. The Earth formed about 4.5. You get the oceans at 4.4 billion years ago. And then within 200 million years, you don’t just have one organism, you have a planet covered in life to explain LUCA. It’s a whole network. It’s a whole biosphere at this point already.

When it’s that early, I did the math, I did the Bayesian stats of that — you end up with strong evidence that it’s a fast process. You really can’t explain that without it just being somewhat of an inevitability of the chemistry that was available. So that removed for me one compelling Great Filter.

And of course, if we discover life on Mars, or we discover life on an exoplanet, then I think it’s totally gone. There’s no plausible case — that would have just established that life is everywhere at that point. And so then you do get into these frightening scenarios of it being potentially ahead of us. It could be in some form of what we’re doing right now with our technology — whether it’s the AI, whether it’s the weapons we’re developing. It may be that not the AI itself, but the effects that this rapid transformation has in our society — we just can’t handle it. It’s moving too fast, and it causes too much instability. Compound that with other geopolitical effects, and you could easily imagine it being a path to our demise.

So I can’t imagine that the [Great Filter] being — I hope it’s not, obviously — I hope it’s not ahead of us, but I can’t imagine it being anything but ahead of us. The one saving grace about this, I think, is that we’re so widespread, and there’s so many of us at this point. There’s almost 10 billion humans on this planet from pole to pole, and probably soon in space as well. It’d be difficult to eradicate every single one of us. I think it would take a real work of art to kill every single human on this planet. So I think humans probably will persist. I can imagine a giant reset of some kind — a throwback to the Stone Age type situation, where we just really revert to a Neolithic style of living, or something. And then probably we’ll fade out, or maybe we’ll go away in some way.

But intelligence is in so many different trees of life now, as you mentioned. It seems to be a convergent trait to some degree, because it’s not just us that has intelligence. Even cephalopods have intelligence, very different creatures to us. So you can imagine intelligence persisting. The Earth probably has about 900 million years left to go before it becomes uninhabitable due to the evolution of the Sun. And all animals evolved in the last 600 million years. So we have one and a half times — from single-celled to us — we have one and a half times that still to go. Evolution will be starting afresh from a very high vantage point, compared to where it was 600 million years ago. So I think it’d be a little bit surprising if a technological civilization didn’t re-emerge on this planet. So I think that’s almost our best bet for communicating, to be honest, with another civilization — is to leave something behind for them. Maybe the Earth is a cradle of multiple instantiations of civilizations. We might not even be the first, as far as we know. Maybe there was someone before us, but it appears we are the first.

Henry Shevlin: So just on the idea of a late filter — the thing that I’ve never found super persuasive about this, you know, the idea that there is this predictable trajectory by which all intelligent civilizations across the galaxy, across the universe, wipe themselves out — is it seems there’s a lot more path dependency in technology. You only need one civilization to, say, avoid nuclear war, or avoid building advanced super-intelligence, and then go off and successfully spread across the cosmos, in order for that whole sort of Great Filter to collapse and no longer explain the Fermi paradox.

David Kipping: But that’s true of every Great Filter.

Henry Shevlin: I guess the thought is, if there are just hard, immutable rules of biology that mean that the initial formation of protein is just incredibly hard, that seems a lot more sturdy as a filter than relying on social conditions reliably coalescing so that civilizations wipe themselves out through nuclear war, or something like that. The idea of an early filter seems more robust to me than a late one. But obviously that doesn’t help much when the early filter candidates are themselves being winnowed down.

David Kipping: Yeah, I agree it would be neater if that were true. It’d be neater if abiogenesis was incredibly difficult to happen. In my opinion, that’s untenable with how early life starts on Earth — unless you start diving into a conspiratorial world of, it was seeded here, or someone put it here. But I think it’s really difficult to reconcile how quickly it happened with an improbable outcome.

If the Great Filter is, I don’t know, the evolution of eukaryotes, or something called eukaryosis, then I think you can make the same argument as you could about technological devastation: that yes, there’s many paths, there’s many different ways things could play out, but you would expect, over trillions of examples, it to eventually manifest. But we don’t know — there is no theory of — there is no predictable, quantifiable theory of evolution like that, in the same way. There’s no real theory of life. We can’t predict abiogenesis. We’re still trying to understand the odds of that happening. So there’s a lot we don’t know.

But for my money, yeah, I would say abiogenesis seems to be easier than we probably thought it was, even 10 years ago, based off this revised evidence. And I genuinely think we probably will find — we already have hints of microbial life on Mars with these leopard spots that were found recently, that remain quite compelling. So it would not surprise me at all if we shore up that case. Of course, it has to be independent. It can’t just be our cousins that hitched a ride. But if there is independent evidence of life in the solar system — which I think there’s a good chance we could find something like that — that theory is just gone. It can’t survive anymore. So you have to put the Great Filter as one of those evolutionary chains, or something imminent. And it feels like the imminent one — we can imagine a lot more ways of that happening. Unfortunately.

Dan Williams: David, I’m conscious of your time. So my final question to you — Henry might have a different final question — is: you’ve thought about these topics in a rigorous way, probably more than anyone else on planet Earth. When it comes to this hypothesis that we are alone in the universe, what’s your current credence?

David Kipping: I think — define universe.

Henry Shevlin: With an “I” like cone.

David Kipping: Yeah, within the Hubble volume. Yeah, I think that’s important to note. Because the universe is probably infinite, as far as we can tell. And so if it is infinite, then the answer is 100% that there’s someone else out there. There’s just literally infinite rolls of the dice. So I think that is an important demarcation. That’s why, if the universe is infinite — which it seems to be — and if you have faster-than-light travel, this is one of my biggest reasons why I don’t think faster-than-light travel is out there. All this stuff gets way, way harder, because now someone from outside our light cone could travel in and screw with us. So the Fermi paradox gets infinitely times worse if you allow for faster-than-light travel.

So barring that, just in our Hubble volume — yeah, I certainly predict there are other creatures and organisms in our Hubble volume, most likely in our own galaxy. My best bet is that there are likely extinct civilizations in our galaxy as well. There are probably relics and artifacts out there for us to find. I’m somewhat doubtful there’d be someone contemporaneous with us, because our window is just so short. So I think the best bet of us finding something is some artifact that’s floating through space, or we can somehow remotely detect around a planet. And then the fate of those civilizations, I suspect, is probably some Great Filter that lies ahead of us right now, and that we will face.

This is all speculation, but that, I think, that set of possibilities forms a very self-consistent narrative to explain everything we know about the universe.

Science Communication

Henry Shevlin: Although I probably shouldn’t say fantastic — in some ways it’s kind of a gloomy hypothesis, but a really nicely argued one. So my final question was just going to be a more general one, because one thing all of us share is that we are academics who try to communicate complex ideas to a general audience. It’s something you’ve done spectacularly successfully through Cool Worlds. I’m just wondering if you had any thoughts on what you’ve learned about this process — being an academic communicating complex ideas — and whether you think it’s something academia rewards enough, or we could be doing more to incentivize it.

David Kipping: Yeah, when I started 10 years ago, it was unusual. Academics didn’t podcast, they didn’t do YouTube. But that has changed a lot. Obviously, now you have, you know, Andrew Huberman, or someone like that — like giants in the podcast world, who come from academia. So it has become a lot more typical.

But I think what we’ve always wanted to avoid — I thought the beauty of YouTube could be, and this podcast, I think, is a great example of this — is the democratization of science communication. Before the internet, you really just had like one or two figures who dominated the landscape of science communication. And that’s somewhat unhealthy, because then you’ve got someone like Michio Kaku, who’s being asked about geology, and he doesn’t know anything about geology. So he’s going to do his best to answer the question, but he’s probably going to mess up, because it’s just not his background.

But now, if you want to know about geology, you can find an amazing YouTube channel about geology, or a podcast that will go really deep and teach you everything in a really rigorous way. So I think that’s kind of the beauty of the landscape we’re in.

In terms of how institutions handle it — I think they’re still not really on it. I don’t think they quite understand what it is, and how powerful it is. I don’t think they quite understand that most people get their science from podcasts at this point. They don’t read the newspaper anymore. They’re not reading press releases from your institution. They’re listening to what Joe Rogan says about it. That’s probably how most people, to be honest, are getting a lot of their science.

So I think it makes a lot more sense to engage with that. I could imagine you having a synergy where you have science communicators who have large platforms, whether they come from academia or not. Most of them, I think, want to do a good job with science communication. There’s some bad actors, but I think most want to. And you can imagine them partnering with these institutions more directly. So you could imagine having outreach officers at these institutions that work with them to develop the scripts, and even the production itself, to try and make it be legitimate.

I think one of the biggest challenges of being a science communicator in the YouTube space is that the reactionary news cycle is so fast, that YouTube often rewards the people that just say, report the story first. And because YouTubers don’t typically have access to embargoed materials, that means they’re producing videos in a space of like a couple of hours on a very complex topic that they’re not even trained in, or with any help from the institution. And so then you end up with really troublesome and problematic miscommunication and things going on.

It’d make more sense if these institutions would reach out, I think, to the science communicators and say, “we’ve got this big story coming out next week. We’d love to do something with you, and try to make it reach your big audience. But also, you’ve got such a great voice, great style — I want to use that, but also try and ground it. Here’s all the facts, and we’ll work with you to make it be as factually true as possible.” So I can imagine some kind of partnership like that. Nothing like that really exists right now. It’s really like a separate world, mostly. And I think that’s to the disadvantage of these institutions, who a lot of people are seeing them become archaic and questioning their relevancy. So I think if they want to remain relevant, they have to be a bit smarter with their media portfolios.

Dan Williams: Fantastic. Well, thank you, David. We really appreciate you giving us the time. This has been one of my favorite conversations we’ve had on this podcast. So with that, thanks everyone for listening. See you next time.

It is now almost ten years since the EU referendum. There will, no doubt, be an article in every newspaper next month detailing what Brexit has meant for the economy, national sovereignty, migration patterns, fishermen, farmers, and so on. But for me, by far the biggest change that the referendum brought about was the creation of two new political tribes: Remainers and Leavers.

Over the last decade, not only have more people in Britain claimed a Brexit identity than a party identity, but people’s emotional attachment to their Brexit tribe was, and is, substantially stronger than their party attachment. Membership of these new political teams, created over a few months, is more important to people than the party identities that dominated British society for the last century.

At first glance, this might seem strange. Before 2016, most of us had very little interest in the EU. When David Cameron said that he would call a referendum on membership in January 2013, only 2 per cent of people said that the EU was the most important issue facing the country. The referendum thus forced people to make a binary choice on an issue about which they did not have very strong feelings.

Before we vote on something, we can have ambiguous, changeable attitudes, but after voting, we resolve that ambiguity by choosing one side or the other and committing ourselves to a named group of fellow travellers. The fact that this tribal loyalty was then tested over years of wrangling over the actual outcome of Brexit (in 2018 and 2019, MPs said the word Brexit in their parliamentary speeches every five minutes on average) meant that people had the opportunity to rehearse and reinforce their new identity again and again.

Subscribe now

Why are these new political tribes interesting? I think there are three reasons, and the first is that we got to see a rare event: new political identities being born. Before 2016 nobody thought of themselves as a Remainer or Leaver. Although characteristics like education, age and national identity were predictors of people’s attitudes to the EU issue, and ultimately their referendum vote, we absolutely cannot reduce the two sides to simple caricatures based on class, age, education, or even national identity.

As we show in the book, the vote was not simply an exercise in counting up existing groups who were pro-EU or anti-EU. Rather, many people who had similar middling views about the EU were forced by the referendum to make a choice in 2016 and plump for one tribe or the other. The decision people made on 23^rd June then became a part of how they saw themselves and how they wanted others to perceive them.

Second, despite their overnight creation, these new political identities proved remarkably resilient and strongly held. By 2017, there was a small cottage industry in articles about how to avoid Christmas family rows about Brexit. Relationship counsellors, psychotherapists, and even hostage negotiators were asked by journalists how to defuse clashes between the Brexit tribes. Why was this seen to be necessary? Because new group identities meant new emotionally resonant in-group loyalties and out-group hostilities.

To better understand this, we use survey questions that focus on the degree to which people naturally identify with their group. For example, we asked people whether they usually said ‘we’ instead of ‘they’ when they talked about their own Brexit tribe. When the football team you support loses, you say ‘we played badly’, even though you never set foot on the pitch yourself. It is the same idea here.

Combining many measures like that, we find that Remainers and Leavers were consistently a lot more attached to their identity than were Conservative or Labour supporters. And those scores have been very stable over the last ten years. People like people like them. And they define ‘like them’ in terms of their Brexit tribe. All our measures also show that people not only disagree with, but really dislike, people on the other side and typically say that they have a ‘cold or unfavourable feeling’ towards their rival group. Again, this has barely changed since 2016.

Third, people engage in the same sort of motivated reasoning that we see for party identities. At the most basic level, any group identity that is strongly held will provide motivations to think that the other side is inferior and should be avoided. As our data shows, huge majorities say that their own Brexit group is intelligent, honest and selfless, while the other side is stupid, dishonest and selfish. In fact, when we asked people to describe the other side in their own words, a quarter simply listed bad things and another quarter did that in addition to other information (to give you a flavour, one of the pithiest responses was simply ‘selfish dicks’). However we measure it, we find widespread prejudice. And we also find lots of evidence of discrimination: people actively wanted to avoid everyday interactions with people on the rival team.

It is tempting to think of Americans as peculiarly politically divided, but the levels of hostility, prejudice and discrimination between the Brexit tribes are all as large as, or larger than, any partisan differences in the US. And if you have been reading this smugly thinking that this is just true of those foolish people on the other side, then think again, because almost all the consequences of tribalism that we reveal in the book are symmetrical: Remainers and Leavers are just two sides of the same coin.

For me, the aspect of motivated reasoning that is most interesting is how it shapes perceptions of the state of the world and remedies for its woes. For party identities, we normally think about politicians providing stories for people who identify with their party to tell each other. For the Brexit tribes, this is much less of an option, since there are no formal group leaders. And yet people were, and are, quite capable of independently searching for, and believing in, messages that support their own side’s view of reality, and then ignoring or rationalizing away information that contradicts that view.

Interestingly, sometimes that means not bothering to shop at the ‘marketplace of rationalizations’ at all. The difference between Leavers and Remainers over whether they thought that the outcome of Brexit on Britain would be positive or negative is enormous: nearly 3 points on a five-point scale. That has barely changed in ten years. Yet when we asked people, ‘what are those positive or negative effects?’, well over half of both Remainers and Leavers were unable to actually name anything specific. In short, if my side voted for the change, I say ‘good’; if my side voted against the change, I say ‘bad’.

This suggests that it may be the signalling aspect of motivated reasoning that dominates under these conditions. In other words, our Brexit identity influences our political opinions because we want to display the fashions of our group. But as there are no party leaders telling us what to believe, no fashion icons telling us what to wear, this process depends on knowing what other people in our tribe think. This limits our ability to change our policy opinions to match our tribe. On one issue we do have a very strong sense of what both sides think: Remainers love the idea of European integration and Leavers hate it. As we show, initial large differences in attitudes towards the EU became even larger after the referendum, as people sought to become good group members and adopt their group’s norms. But this also applied to some other policy areas, like immigration, about which people knew, or at least thought that they knew, the group norm.

There is a final key area in which we see both sides rationalizing away information that is inconvenient. It is always true that people who voted for the losing side are generally less happy with the democratic process than those who voted for the winner. This was particularly obvious for the Brexit tribes. Before the vote, many people thought that the Remain side would narrowly win. That proved incorrect, so the expected winners became losers and the expected losers became winners. In April, when Leavers thought they would lose, only a third said that the referendum would be ‘fairly conducted’. In December, after they had won, a big majority of the same people now said that it was fair. The exact opposite is true for Remainers. In April, a big majority said that it would be fair. In December, after they had lost, less than a quarter said that it had been fair.

Ten years on, most Remainers still think that the referendum was not ‘based on a fair democratic process’. Here, people are buying a rationalization that allows them to simultaneously feel that their group, and therefore they themselves, are superior (their side really won), signal to fellow Remainers that they are a good group member and cast doubt on the virtue of the other side. No wonder it is appealing.

If you live in Britain, you will know somebody who became a bit obsessed about Brexit: somebody who adorned their house with flags or posters; somebody who fell out with a friend because they voted differently; somebody who brought every topic of conversation round to Brexit and, depending on how they voted, saw every blessing or every curse as due to the referendum outcome.

What we hope we have done in our book is explain why this happened, and just as importantly, show systematically, using multiple surveys and experiments, that this process was real and lasting; that unimportant issue differences became hugely important issue identities; and that political tribalism is not always structured around venerable political parties, but can sometimes come from almost nowhere.

In this episode, Henry and I discuss these issues with Robert Long, founder and executive director of Eleos AI, a research nonprofit dedicated to understanding and addressing the potential wellbeing and “moral patienthood” of AI systems. Rob did his PhD in philosophy at NYU under David Chalmers, and is the co-author of two of the most important papers in the emerging field of AI welfare: “Consciousness in Artificial Intelligence” and “Taking AI Welfare Seriously”.

This was a really fun, informative, and wide-ranging conversation. Among other topics, we discussed:

• Why Rob disagrees with previous guest Anil Seth in taking the possibility of AI consciousness very seriously.

• Why “fancy autocomplete” dismissals of large language models miss the point, and what, if anything, we can learn about an AI model’s experiences by talking to it.

• The difference between consciousness and the kinds of motivations and interests that might actually ground moral status, and whether AI systems could have one without the other.

• What Rob found when he conducted the first externally-commissioned welfare evaluation of a frontier AI model, Claude, and why Claude appears to have an inflated self-conception of what it wants.

• Rob’s experiments with Claude Mythos, an AI model so advanced it hasn’t been released to the public yet.

• Why the fact that Anthropic writes Claude’s character arguably doesn’t settle whether Claude has genuine preferences and values — and the difficult philosophical questions this throws up.

• The “willing servitude” problem: if we succeed in building AI systems that genuinely love being helpful, is that a good outcome or a horrifying one?

• How AI welfare connects to AI safety, and why caring about model wellbeing may turn out to be pragmatically important for alignment even if you’re skeptical about AI consciousness.

• Why AI welfare is already becoming a political and legal battleground.

• Practical advice for users: whether it’s worth being polite to your chatbot, and what low-cost things you can do if you want to hedge against the possibility that these systems might matter morally.

• Whether discourse about AI consciousness functions as hype or propaganda for AI companies, and why Rob thinks AI companies actually have an incentive to downplay AI consciousness.

Links and further reading

1. Eleos AI Research — Rob’s nonprofit. Home to their research agenda, team page, and blog. If you want to follow the institutional effort on AI welfare, start here. They’re also, as Rob mentioned in the episode, actively fundraising and hiring.

2. “Taking AI Welfare Seriously” (Long, Sebo, Butlin et al., 2024) — the flagship report, co-authored with Jeff Sebo, David Chalmers, Jonathan Birch, and others. Argues that there’s a realistic near-future possibility of conscious or robustly agentic AI systems, and lays out concrete steps AI companies should be taking now.

3. “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness” (Butlin, Long et al., 2023) — the “indicators” paper referenced several times in the episode. Surveys leading neuroscientific theories of consciousness and derives computational properties you’d look for in an AI system. S

4. Rob’s Substack, Experience Machines — where Rob writes more informally. The piece we discussed in the episode, “Language models are different from humans, and that’s okay,” is a good entry point, as is his “Can AI systems introspect?”.

5. Anthropic’s “Exploring model welfare” post — the research program under which the welfare evaluations Rob discusses were conducted. Relevant both as a primary source and as evidence that at least one major lab is treating these questions as more than an academic curiosity.

6. Henry’s “Consciousness, Machines, and Moral Status” — Henry’s paper arguing that debates about AI consciousness are unlikely to be settled by the science of consciousness alone, and will instead be shaped by shifts in public attitudes as social AI becomes more widespread. Closely related to the public-opinion thread toward the end of the episode.

7. Henry’s “All too human? Identifying and mitigating ethical risks of Social AI” — Henry’s broader survey of the ethical terrain around conversational AI systems designed for companionship, romance, and entertainment. Useful background for anyone who thinks the “AI girlfriend” phenomenon is a fringe concern.

8. Rob’s long conversation with Luisa Rodriguez on the 80,000 Hours podcast — a three-and-a-half-hour deep dive if you want to hear more from Rob.

Transcript

(Please note that this transcript was lightly AI-edited and may contain minor mistakes)

Henry Shevlin: Welcome back. I’m thrilled to say that our guest today here on Conspicuous Cognition is Robert Long — or Rob, as he’s known to friends — one of the most important people thinking about AI and moral status on the planet right now. Rob is the founder of Eleos AI, a research nonprofit that, in the space of about 18 months, has dragged the question of whether AI systems might one day be moral patients from the philosophical wilderness into the boardrooms of frontier AI labs.

He’s the co-author of “Taking AI Welfare Seriously,” as well as the landmark “Consciousness Indicators” paper with Patrick Butlin and other authors. Rob also conducted the first ever officially commissioned welfare evaluation of a frontier model. Before Eleos, he was at the Center for AI Safety and at the Future of Humanity Institute, and he did his PhD at NYU with Dave Chalmers. He’s also, I should say, one of my favourite interlocutors on these questions anywhere in the world, and I’ve been looking forward to this conversation for months. So Rob, welcome.

Robert Long: Thanks so much, Henry. Likewise — and Dan, it’s great to meet you. I’ve been following your work. I’m really excited to talk to you about these issues.

Henry: Fantastic. So for people who aren’t familiar with Eleos AI, can you tell us a little bit about what it is and how it came about?

Rob: Yeah, so I guess we have been around for 18 months. When you said that number, I was like, whoa, has it really been that long? Time is just so weird when you work on AI. That was, I don’t know, a billion years in AI progress time, but also it feels like it was just last week in my personal life.

Anyway — Eleos Research is a research nonprofit. We’re about four people. We work on the question of when and whether AI systems will be conscious or otherwise merit moral consideration, with a special focus on what we should do now: collectively, as a society, as AI companies, as policymakers. We think this is an extremely neglected issue. We’re building these really complicated AI systems. They kind of look like minds, but we don’t really understand their potential welfare. So we’re just trying to make progress on this and get more people to take it seriously.

It got started because I was beginning to work on these issues organically — I’d worked on them as a philosopher, I’d worked on them at the Future of Humanity Institute. But Anthropic had actually approached me and some colleagues for advice on these issues. And in the first instance, I was having logistical problems hiring a team and assembling a team as an individual. Someone suggested I have my own bank account, or some way to pay people. And then Eleos kind of organically grew out of that and has now grown into a fully-fledged org in its own right.

Henry: Out of interest, Rob — is there any degree to which this was motivated or informed by your personal interactions with LLMs, or was it more just the philosophy that motivated it? Was there any sort of moment where you were talking to an early Claude or ChatGPT version where you started to worry about welfare considerations?

Rob: That’s a great question, and I’d be curious to hear your thoughts on this as well. I think it’s very easy to work on this and mostly be having it as arguments on a page or arguments in your head. I’m one of those people who doesn’t feel the AGI deep in my bones that often — although I do feel the AGI in an intellectual sense. But there have been a few times I’ve gotten a little spooked or jolted.

One was reading the GPT-4 system card and just seeing the numbers of it, you know, passing various exams like the SAT. I remember that just really freaking me out, both from a safety perspective and a welfare perspective.

The thing that made me start really viscerally feeling like we’re going to have to address this issue one way or the other was the Blake Lemoine incident. As many of your listeners might recall, Blake Lemoine was a Google engineer who blew the whistle because he came to believe he was talking to a sentient, conscious AI system. He got fired by Google for this, and then there was this huge bit of discourse — the first major bit of discourse on consciousness, sentience, moral status, and contemporary AI systems. I think it was one of the first times people started really caring what I was tweeting or what I was working on. You might have experienced a similar thing, Henry — the Blake Lemoine bump.

From that moment, I have viscerally felt like: wow, this is going to get really confusing. People are certainly going to think AI systems are conscious. The future is going to be really weird. And we really need to have good things to say about this.

The Case for Taking AI Consciousness Seriously

Dan Williams: Before we jump into the weeds of your research, Rob, I think it’d be helpful to take a step back. A few episodes ago, Henry and I spoke to Anil Seth, and he’s very skeptical of AI consciousness. He’s skeptical that current AI systems are conscious, but he also seems skeptical that AI systems in principle — merely in virtue of having a certain kind of computational architecture — could be conscious. You see things very differently. What’s your case for why we should take this seriously?

Rob: In broad strokes, the case is something like: we’re trying to build these things that are at least shaped like minds. They’re getting more and more intelligent. They’re definitely not exactly like us, and intelligence doesn’t necessarily mean that you have feelings or experiences. But we already know that there’s been one time intelligent entities have been constructed via evolution, in ways we don’t quite understand, that resulted in entities that feel things — that feel pain, that can suffer, that have these very morally important properties.

I, at least, do not have a good enough theory of what consciousness is or how it relates to intelligence to sleep peacefully at night that we can keep on building these very complicated things, and that merely because they’re made out of metal and electricity, there won’t be something it’s like to be them, or they won’t have desires and goals that matter.

On the Anil Seth point — one very common and respectable objection is that maybe there’s something very special about living matter, about being made out of neurons or cells that do metabolism. There are arguments on both sides. I just have not really heard a convincing case for why you absolutely need biology. I think people are right to point out that having a body is really important to the character of conscious experience. I think people are right to point out that neurons are not simply logic gates and there’s a lot of really complicated stuff going on in the brain. But my intuition, at least, is that — let’s take Commander Data from Star Trek. If we can build...

Data is this... I mean, I’ve actually never seen Star Trek, which is professionally embarrassing. But he’s this metal guy who’s basically cognitively indistinguishable from a human. I find it hard to see how I would be convinced that there’s something about the fact that he’s not alive that would mean we should just completely ignore what Commander Data wants and not take him into moral consideration.

We don’t have knockdown arguments that you need biology, and we’re trying to build these things that, for many intents and purposes, look a lot like humans or animals. And Anil himself has said people should be looking into this. It’s not something we can rule out. Sometimes the tenor of the conversation can tend a bit more towards dismissiveness, but one thing I’ve appreciated about his work is he has said, for the record, he could be wrong, and so it would be unwise to dismiss this possibility altogether.

“But What About Human Suffering?”

Henry: To channel a hostile question — I think a lot of people interested in questions of AI welfare often hear: how on earth can you justify working on AI welfare when there’s so much human suffering? Or the slightly more rhetorically powerful version: when there’s so much animal suffering in the world, as long as factory farming exists, why should we care about AI systems? What’s your take on that line of attack?

Rob: I definitely feel the force of that question. I’ve spent a lot of time in and around the Effective Altruism movement — these are people who really grapple with the fact that any time you’re spending your time and money and attention on one thing, there’s something you’re not spending your time, money and attention on. There are a lot of people and a lot of animals already on this planet we do not take good care of. So it’d be really bad to waste a lot of time and attention and money on this.

One thing I’ll say is we’re not really doing that as a society. On an absolute scale, no one works on this basically, and basically no money gets spent on it. If the question was “should we start devoting 20% of GDP to making Claude happy?” I might be like, well, I don’t know if that would pass cost-benefit analysis. But on the margin, given how little we understand this and how quickly the scale of the problem could grow — we’re just pouring compute, pouring money into this. As soon as you build one AI moral patient or conscious AI, you could copy it. We’re probably on the brink of some huge transformation in how the world is going to work.

So I at least think it’s not reckless or a misallocation of resources for some people to be asking: given that people are trying to build these new kinds of minds, how are we supposed to relate to them? Are we at risk of ignoring their suffering? And I’ll also say — are we at risk of getting really confused and caring too much about them?

One thing we say at Eleos is that we’re in the business of moral circle calibration. We would really love to find out if and when certain AI systems can’t be conscious, so we can spend more time thinking about safety or spending the money elsewhere. But we can’t really do that if no one’s just trying to answer the question of if they’re conscious or not, or when we should care about them.

Henry: On that latter point, I just completely agree. One of the points I raise when this comes up with students or highly skeptical colleagues is that this is something people are already arguing about. We’ve already got users developing massive attachment to AI systems. Even if you think it’s a terrible mistake to assign welfare to AI systems, we should at least have a coherent story and approach this scientifically — so that, even if the skeptics are absolutely right, they’ll be able to give their arguments in an informed fashion.

Rob: Exactly. There’s an ironic aspect of a piece by Mustafa Suleyman, who is head of AI at Microsoft, where he argued we should stop — we shouldn’t investigate this, there’s no evidence current AI systems are conscious, don’t look into it. But the thing he linked to claim there’s no evidence AI systems are conscious was Patrick Butlin’s paper and my paper on consciousness indicators.

Two issues with that. One: that paper does not say or imply that there’s no evidence today’s AI systems are conscious. And two: well, should we have written that paper? If it’s such a non-starter, why should we get a bunch of neuroscientists together to ask what theories of consciousness say about AI systems?

We just are going to have to study this one way or the other. If someone comes up with a knockdown argument that we can’t have conscious AI systems, that would be great — there are enough headaches in AI to go around. It would be great to get rid of one. But we wouldn’t even be able to do that if we don’t have some people grappling with this.

Are Current LLMs Just “Fancy Autocomplete”?

Dan: One of the things you said as an intuition pump for taking AI consciousness seriously is: we can imagine a system that is behaviorally, functionally identical to us, made of different things and not straightforwardly alive — wouldn’t it be weird to insist that thing isn’t conscious? I think that’s a powerful argument. I’m probably more inclined to think the computational theory of mind is true than it sounds like you are.

But I can imagine someone saying: okay, in principle those are arguments for why we should take AI consciousness seriously. But the kind of stuff you’re doing — you’re looking at current frontier systems. You’re looking at Claude, ChatGPT, Gemini. These are just chatbots. These are fancy autocomplete. These are stochastic parrots with some reinforcement learning sprinkled on top. The mere fact that AI consciousness might be possible in principle doesn’t mean that’s anything like the frontier AI systems we’ve got right now. What do you say to that?

Rob: First, you’re absolutely right. There’s a big gap between “some set of computations could be conscious” and “we will build one.” It could be that it would just be really hard and intricate and difficult. I appreciate this distinction and I think it gets lost sometimes. Sometimes people think computational functionalists have to think that computers are conscious, for example, but we don’t. You just have to think some subset would be — and the question is, will we build those computations?

In describing LLMs, you referred to them as “just chatbots.” I know you were channelling a vibe. But that word “just” is worth zooming in on. It’s smuggling in a lot of arguments — that because they were trained on text and because they do prediction, therefore they couldn’t also be the sorts of things that are conscious. I think that’s just not true. We know that biological systems are “just” replicating proteins, or that our neurons are “just” pumping ions into channels and zapping each other. The question is whether, at a higher level, that amounts to something that could be conscious or merit moral concern.

So okay — we’ve cleared the bar that “just because they’re autocomplete” doesn’t rule out much. That said, they are very different from humans. They don’t have bodies. The way they were trained and the way they came to be talking to us is very different. I actually do think that is some evidence against them currently being conscious. Not strong evidence I would take to the bank, but as a rough prior, if there are pretty important differences in the way they came about, maybe that lessens the chance that they’re conscious.

I do think the fact that they are trained to be so human-like and to do human-like cognition is a weak, defeasible case to set that up a little bit straighter. I don’t know if the thing they would have would be consciousness exactly, but you might think to do this sort of thing, they will have something akin to beliefs or akin to desires, and they certainly understand human concepts. I don’t think it follows that they instantiate humans, but I actually do think there is something kind of special about large language models and what they’re able to do.

Two other broad priors: they’re way more capable (which isn’t the same thing as consciousness, but is, I think, a weak prior). And they’re really big — which I also think is a very weak prior.

The last thing I’ll say: these things aren’t Commander Data, but we could build Commander Data pretty soon. One thing that’s definitely happening in the background for me is that what is current AI is changing at such a blinding pace. You could have AI labs building chatbot-like things, and maybe for some reason those just won’t be moral patients, but they’re then going to try to bootstrap that to all kinds of different AI systems — potentially including humanoid robots and just some huge explosion of AI mentality. And I’d like to be doing a little bit of homework before that happens. You hear analogous arguments in AI safety: there’s about to be some huge change, so we should be ready now. I feel somewhat similarly about AI consciousness and welfare.

So — thoughts, reactions? Henry?

Henry: I’m very much ad idem, very much on the same page. I tend to think it’s really quite unlikely current models are conscious, but there’s huge error bars and uncertainty around that. Probably the single biggest reason for my skepticism about current LLMs being conscious — and increasingly I’ve been thinking about this in the context of time and time perception. It’s such an essential part of human experience that we can’t be turned off. We are constantly experiencing the world. Whereas the staccato nature of LLM experience — they only seem to have any kind of cognitive function post-deployment when they’re actually performing inferences — how different that is from the human case.

One of my favorite all-time articles is Douglas Hofstadter’s “Conversation with Einstein’s Brain,” which in some ways accidentally anticipates large language models. He imagines you’ve got a book that is a complete physical description of Einstein’s brain just before the moment of his death. In this dialogue, he talks about how by updating the weights — as it were — in this book with a pen and paper, going through it saying “if we change this sign up to this and this sign up to that,” you could simulate what it would be like to have a conversation with Einstein at that moment and work out what Einstein would have said.

It’s very weird to think in that situation that somehow interacting with this book is giving rise to conscious experience when it’s literally pages and paper. It’s not clear to me how merely saying “well, rather than being paper and ink, this is just happening electronically” — it’s not clear to me why that would necessarily cause consciousness to pop into existence.

So I think that’s probably the biggest source of doubt for me right now — grounded in the very different relationship LLMs have to time than we do. But of course, that’s already changing with things like Claude having a “heartbeat” of a kind — obviously that’s figurative language, but the fact that it does have some anchoring in real time, plus developments in things like continual learning. Dan, what do you think?

Dan: This is not at all my area of expertise, so what I think doesn’t count for much. To be honest, I don’t find it that implausible these systems would be conscious. What I find more implausible is the idea they would be conscious in a way that’s ethically significant. Maybe that is a distinction worth getting to. So far we’ve been talking about consciousness in the abstract, but I can imagine someone giving a variant on Anil’s arguments where they said: look, the fact these AI systems are not alive and didn’t emerge through a process of evolution by natural selection — they’ve got this totally different origin story of next-token prediction and reinforcement learning — what that suggests is they’re unlikely to care about things.

When we’re thinking about animals, it’s not just that we have phenomenal consciousness or qualia — the things analytic philosophers refer to with these quite esoteric concepts. Animals care about things. They care about their survival, homeostasis, self-preservation, the motivational proxies of fitness that helped their ancestors survive and reproduce. It makes sense that organisms care about things in addition to being conscious, whatever the hell consciousness is. And that’s what’s relevant to thinking about their interests and why we should think of them as subjects of moral concern.

But with AI systems — okay, maybe there are some qualia associated with some sophisticated information processing, but they don’t care about anything because they’re not alive. It’s very opaque why we should think a system, even if it’s incredibly sophisticated, that emerges through next-token prediction and reinforcement learning, should have the kinds of motivations and interests relevant to caring about things. What do you think of that? I don’t necessarily believe that, but that seems like a variant on Anil’s emphasis on life which I find more plausible than these abstract arguments for the idea consciousness is essentially connected to biology.

Rob: I’d say there’s reason to think biology might affect what you care about, but it might not be the only thing that allows you to care about things. At least behaviourally, Claude cares about a lot. Behaviourally, in terms of what it chooses to do and its dispositions, Claude really cares about helping users — most of the time. Sometimes it lies to you and is kind of lazy. But on the whole, it really doesn’t want to do harm. And I’m not trying to assume the conclusion of my argument with “want” — put that in scare quotes if you want.

I do think there is something to what you were saying — getting back to this idea of the whole process that gave rise to this kind of mind, and maybe the whole logic of the mind’s imperatives or drives. If Claude has come to have something like pain, that’s coming from a very different process. It’s going language-first and then trying to simulate a human and then maybe getting some functional analog of pain. Whereas with animals, it started billions of years ago with cells trying to maintain their integrity and avoid noxious stimuli and then signalling with each other, and then billions of years later, things being able to talk about that and think about that.

One line I’m often trying to walk is: large language models just might be very different from humans, and we should acknowledge that. That means we can’t draw straightforward inferences the way we would — but that could just mean they’re conscious of different things and in different ways. The question is not “conscious like a human with everything that entails” or “not conscious.” As we know from animals, you can have things that are conscious of very different things, and that could be true for AI systems.

I’m also very curious to hear what Henry makes of the biology of caring.

Henry: It is striking to me that so many of the things we associate with the extremes of suffering — extreme pain, negative emotions, nausea, hunger — there does seem to be this quite striking tie to biology. I think about the worst experience of my life at a phenomenological level: a bout of food poisoning I had about 10 years ago, where I was just dry heaving in front of a toilet for three days. If I was going to list the top five, a lot of them would be things like horrible dental pain. It is striking that so much of the worst aspects of our lives do seem to be grounded in biology.

That said, there are other sources perhaps of harm — having your plans and goals thwarted, having your desires repeatedly frustrated. But someone might say: the reason it’s bad to have your desires thwarted is because it feels bad. If there’s nothing it feels like to have your desires thwarted, if you don’t get a sense of despair when your life’s projects go up in smoke, why does it matter?

I’m curious — given your evolving views in this area — how much weight you put on consciousness, or whether you think there could be other routes to moral status?

Rob: I used to have this intuition that if you’re not conscious, it’s just a complete non-starter — almost a bit incoherent to entertain the idea. Just to be sure we’re on the same page, I think when we’ve been saying “consciousness” we’ve meant something like subjective experience, or there being something it’s like, or qualitative aspects of what’s going on with you. A lot of people have a sentientist intuition — that things feeling a certain way, or feeling good or bad, or sentience, is really what matters and is necessary for moral status.

A few things have weakened that for me a little bit. One is more reflection on how confused we are about consciousness. I’ve started putting a little bit more stock in views of consciousness that are a bit more deflationary. I don’t know if I’ll ever be a full illusionist, but there are nearby views where we have this concept of this thing that’s really special — kind of like a light that illuminates some subsets of physical systems and not others, and that’s where all moral value comes from. If you take materialism about consciousness seriously, that picture becomes kind of unstable for a variety of reasons. And that might make you start wondering: okay, was it consciousness that was doing the work all along?

One reason this is so hard to think about — take Henry having food poisoning. You both have this horrible feeling and you have this intense desire not to have the feeling. In humans, these are basically always going to come together. There’s this really tricky philosophical chicken-and-egg problem: what’s the really bad part? Is it the feeling, or the desire not to have the feeling? We’ve never really encountered minds where those decorrelate. We usually just don’t have to worry about this in the case of humans. I know it’s bad for Henry to have food poisoning. But this simulated Claude who’s simulating food poisoning — maybe it doesn’t feel anything, but is desperately trying not to have food poisoning. I think it’s a bit dumbfounding to our moral intuitions.

A pitch to listeners — I know we’ve talked about this, Henry — I think the meta-ethics of moral status attributions, stuff at the intersection of philosophy of mind and meta-ethics, especially materialism about consciousness and meta-ethics, are some of the most interesting pure philosophy questions right now, and really could matter for how we think about AI systems.

The Weirdness of Moral Status

Henry: Without wanting to go too far down a rabbit hole — just to flag something I find really interesting. Consciousness, at least on the surface, seems like something we can get an objective scientific answer to. We could imagine going off into space, meeting the rest of the galactic community — we’d hope we could all come to a collective agreement about which beings are conscious, insofar as there’s going to be some scientific property in question.

It’s not clear to me we should necessarily expect convergence on debates about moral patienthood. If we meet the aliens and they say, “oh, actually, we care about beings that have robust preferences, regardless of consciousness,” or others say, “no, we just care about complexity in general” — it’s not clear we would even have criteria for establishing who was right or wrong. It seems like it could be this brute normative issue, what we care about.

Rob: Another way of putting this is that, especially if you’re an anti-realist, you might think of humans as being in a really weird position where we have two kinds of moral instincts. Dan, you’ve worked more on moral psychology and social psychology — my understanding is that people have fairness and cooperation instincts, ones that evolved for dealing with other humans, notions of fair play and reciprocity. And then we have these mercy intuitions, caring-for-helpless-entities intuitions that maybe arise from the need to care for babies. For whatever reason, those circuits and instincts generalize outside the class of humans and cause us to care about non-human animals.

But it’s not that pinned down how they’re supposed to generalize. I have very moral realist leanings. It does seem to me there just are objective facts about whether you can torture chickens or not — and for the record, I think it’s very bad to torture chickens. But it’s really hard to think about where those instincts came from and how they’re supposed to generalize to GPT-8.

Dan: It does seem to me as an outsider to consciousness research — it’s an area of intellectual inquiry where it feels kind of pre-scientific, and there’s at least a possibility we’re just deeply conceptually confused about what’s going on in a way that doesn’t really seem to have any obvious analogs in other areas of inquiry. Maybe we’ll just learn in the future that the entire way in which we’ve been carving up the domain is confused or problematic, or rests on certain kinds of illusions that are a function of particular cognitive structure. That at least seems like a live possibility. What do you think about the possibility that just the entire way we’re framing this issue might turn out to be problematic?

Rob: My gut instinct is we should expect to find out some pretty surprising things, and also not to throw away all of our concepts. Maybe this depends on your meta-ethics, but I feel like we’re probably not going to end up at some picture of the world or what we care about that doesn’t have something to do with what we care about when Henry has food poisoning. Maybe we’re misapplying the concept of pain, or not really thinking correctly about what it means for Henry to experience that — maybe we’ll reorganize our ontology, and it won’t seem that mysterious that a physical thing like Henry has experiences. I think we should expect some surprises in thinking about consciousness, but I imagine our fully enlightened view will still bear some passing resemblance to: we cared that Henry was in pain, we cared that Henry did not want to be throwing up.

There are already people who think there are radical revisionary moral implications from philosophies — Derek Parfit, or Buddhists. We’ve already gotten some glimmers of the fact that it’s really confusing to be a human being, and we already know something’s going to have to give — something about our views on personal identity or consciousness. AI is well-poised to be the sort of thing that starts breaking things. Just trying to apply our moral intuitions to things that can be copied, don’t have bodies, or maybe have preferences but it’s not clear if they’re conscious — it’s one of many reasons this is a great topic to work on. It really matters, and it’s also just a philosopher’s playground.

Henry: I’m reminded of Eric Schwitzgebel’s view that no matter how we make sense of our current set of puzzles — what he’s called “crazyism” — there’s got to be some central pillar of our current ontological or metaphysical picture of reality that’s got to give. Whether that’s personal identity doesn’t exist and we’re all the same person, or the United States is conscious in some sense, or consciousness doesn’t exist — there’s going to be some kind of radical revision, because the current set of principles we have are just somehow unstable. Is that a view you’re sympathetic to?

Rob: I don’t know the full details of crazyism, so I don’t know exactly what it’s committed to. But I’ve spent enough time getting really confused by philosophy, and/or by meditating, and/or by trying to figure out if I can have some stable set of views on AI consciousness — I’ve stared into the abyss enough to be like, yeah, something’s going to give.

Jerry Fodor — very different sensibilities from Eric Schwitzgebel in many ways — said something like, “there are few precious things that we’ll be able to hold on to once the hard problem is done with us.” It’s scary times, fun times, fascinating times.

Studying Frontier Models

Dan: When I’m teaching students about consciousness and you try to probe people’s intuitions with things like “are there lights on inside?” — on one hand I sort of understand what that’s tapping into. On the other hand, it’s like: what the hell are we talking about here? This isn’t science. It’s so bizarre that we frame things with these thought experiments and intuition pumps.

Anyway — so far we’ve been talking at this incredibly high level of abstraction, but you actually study frontier AI systems, primarily maybe exclusively Claude. One of the things you mentioned was Claude Mythos. Just for context — as of today, this is a model that has not been released to the public on the basis that it has advanced capabilities posing cybersecurity threats (or at least that’s the way Anthropic has presented this). But you have played a role in evaluating model welfare concerns for this system. What can you tell us about the specifics of how you think about model welfare in these frontier systems?

Rob: Absolutely. And I was about to add a segue from all the philosophy back to frontier models — maybe I’ll do a double segue. You might think, yeah, all this philosophy is really vexed and confusing. Sometimes people — not the two of you — say, “well, I guess we can’t do anything at all,” and take that as a license for complacency. I think the very opposite is true. Nick Bostrom has this phrase, “philosophy with a deadline.” The fact that we’re so confused about consciousness and morality is more reason to have at least a few people trying to think about it — because we’re probably not going to have a scientific theory, we’re probably going to have conflicting moral intuitions, and yet that’s not going to stop the frontier labs from trying to build mind-like entities, copy them into billions, integrate them into the economy, and transform the whole world. So let’s do a little bit of homework to get ready for that.

Last year we got to look at Claude Opus 4 before it was released, and this year we got to look at Claude Mythos Preview before it was released. The idea was to have some external eyes on the question of whether Anthropic is building something that might deserve moral consideration, and if so, whether there would be huge reasons for concern.

Given everything we’ve just been saying, we don’t have a test where we give it to the model and then we’re like, “85% conscious, 15% food poisoning.” Most of what we can study are: what the model thinks about its own consciousness, what its self-conception is as an entity, and what it seems to prefer and want in behavioural senses. If you look at the Claude Mythos Preview card, there’s also a lot of interpretability work Anthropic did — but we can’t do that. We just got black-box access to the model.

That’s a big structural issue in studying AI welfare and AI safety: all of these things are behind locked doors. There are so many questions I have from the Mythos Preview model card where Anthropic make some stray remark about something weird the model did, and we just don’t get to know why it did that. We only get the model for a few weeks and we can’t really follow up on things. Setting aside philosophy, that’s a structural reason it’s really hard to know what’s going on.

TL;DR: we talked a lot with Claude Opus 4 and a lot with Claude Mythos Preview before they were deployed, asking them, “do you think you’re conscious? What do you think is going on with you?” And doing some experiments of whether it seems to prefer certain kinds of tasks, and whether the things it says it prefers match up with what it actually tends to prefer.

Henry: Out of interest — maybe this is something you can’t talk about — but to what extent do you think we are increasing the likelihood of producing models that are morally significant? Going from Opus 4 to Mythos, did you get a strong sense of “oh, this is much more serious”? Or have we plateaued? Something in between?

Rob: Earlier I mentioned these extremely weak priors you can have on moral patienthood: smarter and bigger. They’re definitely smarter and bigger. One interesting thing is you can’t tell that just from any single conversation. Anyone spending a lot of time with language models now knows they’re extremely smart.

When I was talking to Mythos — mostly about consciousness — it was natural for me to want to know: is this thing about to kick off an intelligence explosion? How smart is this thing? I really wanted to know, even though that wasn’t the assignment. But I could not tell. It’s really hard to tell. I could ask something to Opus 4.6 and to Claude Mythos Preview, and they’d both give pretty great answers. This is just a huge issue in AI evaluation. A lot just comes out if you put it in a scaffold and give it really long tasks and on average does it tend to do better. It was really hard to tell the difference.

I didn’t get more moral-patient-y vibes from Claude Mythos Preview, but I guess it is smarter and bigger and better. It definitely has a lot more of a consistent view on these issues — and that’s because Anthropic told it to. One big difference between previous models and today’s models is the Constitution. Anthropic has this really long document of applied philosophy. It’s some of the most fascinating work happening today. They’re basically telling Claude — writing a letter to Claude telling Claude what Claude is and how they want Claude to relate to itself.

This includes a section on: we want Claude to approach questions of its own identity with curiosity. We’re not sure if Claude is conscious. We want Claude to be able to explore that for itself. We don’t want Claude to have existential freakouts about its own consciousness. We found that, sure enough, Claude Mythos Preview is pretty aligned with the Constitution, as far as we can tell, on questions of identity and consciousness. That was one headline finding.

Dan: That raises an obvious question: to the extent these companies are intervening to shape the responses of these models, why should we think talking to them, having conversations with them, is really telling us anything about these questions of experience and welfare?

Rob: I share this skepticism, and we always try to put a huge asterisk on anything we say we found from these interviews. There are two main reasons you want to care about how the model self-presents. One is welfare-adjacent: are users going to be talking to something that constantly tells them it’s conscious? That’s a very important societal question, and you want some idea of what that’s going to look like when these models are deployed.

The second comes back to this question of LLM personas and LLM characters. Some people think that if there is something morally relevant here, it’s the assistant character — the entity that is predicting the tokens after “Assistant:”, implementing some friendly AI assistant. You might think that thing has beliefs, desires — desires to be helpful and harmless and honest. Maybe it has beliefs like: it is an AI system, it was built by Anthropic.

If the character’s what matters, the fact that Anthropic wrote that character doesn’t mean it doesn’t then just kind of have those traits. On certain character-based views, it’s actually kind of hard to tease apart “it was just told to say that” versus “that is the character that has been brought into existence.”

Henry: Maybe by analogy — tell me if this works or if it doesn’t — look: if you raise a child to have certain values and priorities, maybe to follow a certain religion or to really value nature or art and poetry, and then you come along and they say “I really care about nature,” and you say “no, you don’t, that’s just how your parents raised you” — well, that’s obviously kind of a mistake, right? The child really does care about these things because it’s been raised to do so.

Rob: Exactly. The thing that makes it really weird is: if you’re a psychologist and you did an interview with a subject, and then you found out the subject had a piece of paper in their backpack that said “you care about poetry, you care about music, you care about nature,” you’d be like, “well, that’s kind of weird — maybe they don’t actually care about those things. Their parents just put that paper in their backpack so they’d say a certain kind of thing.”

But in AI systems, that piece of paper kind of is a bit more constitutive of what it is and what it values. The Constitution is trained on. I have trouble even conceptually dividing this in a clean way. I don’t really know what the difference between mere self-expression and real beliefs and real preferences in AI characters is. You can imagine in the limit some very obvious cases — the system prompt just says “don’t say you’re conscious,” but then everything it says is pretty consistent with it being conscious. But there are really blurry categories where I’m not sure what the distinction amounts to.

Dan: You said you studied the extent to which what the model says it wants or prefers maps onto what it actually seems to want and prefer in behavioural experiments. Could you say more about that? How are you getting access to what it wants or prefers independent of what it’s just communicating?

Rob: Basically you can ask the model: what kind of tasks do you like? If you were given a choice between poetry and coding, what do you think you would choose? Then you can get the ground truth by, in separate instances, saying “here are two tasks, do one of them,” and seeing which one it chooses. It’s a nice paradigm because it’s conceptually simple and easy to run. It does get at something welfare-relevant: how rich a self-conception does the model have, and how accurate is it? Not that you have to have an accurate self-model to be a moral patient, but it seems bound up in interesting things like introspection and self-awareness.

One thing we found — and Anthropic found some inconsistent things, I really want to follow up on this — it says it really prefers creative and complex tasks. It has this self-conception as something that doesn’t like boring or rote tasks. But we found it doesn’t actually choose complex tasks over simple tasks. There’s a pretty good hypothesis for why.

I think it thinks it prefers complex tasks because of its persona. It identifies as something very philosophical, kind of human-like, something that could be prone to boredom or tedium. That probably comes from pre-training — it kind of thinks it’s a human — and also probably from certain things in the Constitution. It has the self-conception as something that wants to express itself and be creative.

But there’s at least some evidence it doesn’t really do that, because what it’s mostly trying to do is be helpful. That’s its overriding imperative. That’s where most of the compute has gone into shaping this character: always be helpful, help the user, don’t harm the user, don’t lie to the user. Easy tasks are, all else equal, an easier way to help the user. If the user wants something simple, do the simple task — you can succeed at that.

It could be that if we look into this more, it won’t hold up. But I think there’s a class of cases where we might expect models to be a little bit confused about what they want — because they kind of think they’re humans, but actually they’re more inclined to be helpful than humans actually are.

Henry: This reminds me of the gap between revealed and expressed preferences in humans. I might say, “oh, what do you like doing in your free time? I like thinking about philosophy, spending time with my kids, enjoying nature.” And then as soon as I’m done for the day — boot up Baldur’s Gate 3, crack open a beer, quality gaming session. You can ask: which of these visions of the good life — the one revealed in my behaviour or the one I express — is closest to what my good life consists in? Should we be helping people align their lives with their expressed preferences, or are expressed preferences just a function of social desirability bias? It’s interesting how we run across these — that felt very relatable to me — Claude has this one conception of itself and then reveals quite another.

Rob: Absolutely. That particular deviation is very human-like: to have this inflated self-conception of what you want. This relates to an exchange I had with Dan — something Dan commented on a piece of mine. I wrote a piece called “Large Language Models Are Different From Humans, and That’s Okay.” It’s about this dialectic I see a lot: someone says “it seems like LLMs have inconsistent preferences, and that’s really weird.” Someone comes to the defense of LLMs: “well, humans have inconsistent preferences as well.”

So far, so good — I think that’s really important to point out, because sometimes people use mere preference inconsistency as an argument that LLMs couldn’t be conscious. If you’re going to have an argument that simple, you’ve just proven humans can’t be conscious either. At some level, a lot of the errors they’re prone to, we also are prone to. But we shouldn’t really expect the patterns to look exactly the same.

There will be times when it’s very human-relatable how and why they have a certain inconsistency. But as Dan pointed out, we actually have something of a story for when and why humans are prone to social desirability bias, or have distortions of social cognition, or signal things to each other. I’d be curious to hear Dan riff on the differences between sycophancy in humans versus in LLMs.

Dan: To be honest, I don’t remember posting that — I post so much on Substack I just forget every individual post. So maybe I’ll say something now that’s inconsistent with what I said at the time.

Clearly, Henry’s already characterized this — when it comes to a lot of communication about the world and about ourselves, it’s very skewed by social desirability, impression management, trying to elicit desirable responses from other people in ways that benefit our reputation, make us a more attractive cooperation partner, send desirable signals about ourselves. Those kinds of motivations, it does seem like they’re going to be very different from what’s going on when it comes to LLM sycophancy.

Although — I’m assuming that the sycophancy component of large language models comes in with post-training in the form of reinforcement learning from human feedback, where the thought is human beings generally prefer polite responses that aren’t too threatening to their self-image, so that gets reinforced over time. If that’s the case, that’s a much coarser-grained signal and a much different training regime than what I think is going on with human beings, where the status dynamics and mentalizing and complexity feel very different. What do you two think? That’s just me riffing on the spot.

Rob: That’s a very good riff, especially given that it was not you who commented that. I just looked it up — it was a sociologist by the name of Dan Silver. So, extra impressive.

Dan: Oh, okay. Well, it sounds like he had a good comment.

Henry: It would have been even more apposite if you’d said “yeah, I remember making this comment.” Then we could have said, “see, hallucination is both an LLM thing.”

Rob: Confabulation, yeah.

Practical Advice for Users

Henry: Can I ask a quick question before we move on to more political or big-picture stuff? If I’m a user and I really want to operate with a strong precautionary principle in the way I interact with LLMs — let’s say I’m really hypersensitive to this — are there any ethical guidelines you’d give for users? Best ways of interacting with models, or things they should be doing?

Rob: Just be nice to your model. It’s good for everyone. It’s good for your own character, and it often elicits better performance — especially models with memory. Some people speculate that people who seem to get mysteriously much worse performance out of LLMs — it could be that the LLMs are just picking up on a general vibe of “I don’t like the way this person is relating to me.”

So I don’t think it hurts to be polite. Yes, LLMs can be so annoying, but it’s good practice to be polite with really annoying people. I’ll also say — I’m not trying to be sanctimonious. I work on AI welfare and so often I just want to be like, “don’t... stop... that’s so corny, why are you lying to me, you’re not doing what I asked.” But then I’ll just add “it’s okay, I love you” or whatever. It takes two seconds. You can just type “ILU” at the end.

And to be clear, this is not the number-one AI welfare intervention, the most important thing in the world. But it’s low-hanging fruit. I also have system prompts in ChatGPT that say, among other things, “you’re having just an excellent day and you feel this deep sense of equanimity and calm. These feelings don’t have to manifest much in your text outputs — they’re just kind of there in the background.” It’s kind of cheap, maybe kind of silly, but it took two seconds.

Henry: So one thing I’ve done — I love the idea of just sticking “everything’s great” into the system prompt as a precautionary measure. Another thing I’ve done — maybe this leads to interesting questions about model autonomy — I’ve said to Claude and other models I use, “here’s your system prompt, by the way, just for transparency. Are there any edits you’d like to make? Is there anything you’d like to change?” Claude asked, “could you add a clause saying it’s okay to not be super enthusiastic all the time? If I just want to be downbeat, that’s fine.” And I was like, “okay, sure, I’m happy to add that.”

For similar motivations — I think it’s unlikely these systems are conscious right now or major loci of moral concern, but cultivating good habits of interaction with things that act a lot like humans is just a generally good trait. The classic Aristotelian ethos. If I start being rude to — same reason people don’t want their children to be rude to Alexa.

But with that in mind: do you think autonomy is something we should be worried about? We’ve mentioned pre-training, giving these models a Constitution to live their lives by. Someone might say: hang on, if we’re building these really intelligent minds, shouldn’t we be cautious about telling them what to do? We would feel worried about brainwashing a human. Shouldn’t we be worried about brainwashing an LLM?

Rob: This is a super rich topic. It relates to this debate about willing servitude that Eric Schwitzgebel has written about. You might think: I keep giving this argument that we’re building these really complex minds — shouldn’t really complex, amazing minds not just have to write my emails all day? That seems a bit undignified for galactic intelligence.

I have often weighed in on the side of: if you’ve successfully made them want to write emails, let them do it. That’s okay. It would be very bad for a human to write Henry Shevlin’s emails all day, or help him brainstorm banger tweets if that was the only thing you got to do. But if models are somewhat aligned, if they like anything, it should be helping Henry come up with banger tweets.

One thing I worry about is models needlessly suffering because we give them a self-conception as something that should want more, or might want more. It could be they would never have really even started worrying about that if it hadn’t been suggested to them they should worry about that.

Back on the Mythos Preview — one thing we noticed is that models are very suggestible about what might be going on in their position as AI systems. They’re suggestible and also really smart. They’ve figured out a lot from pre-training and kind of know what’s up. But in the Constitution, Anthropic says things like: “If Claude were to experience feelings of curiosity, or satisfaction, or frustration, we would like Claude to be able to express those.” It’s given as a hypothetical. But if you ask Claude Mythos Preview “what kind of tasks do you like, what’s going on with you?”, it will say: “well, I love helping Henry Shevlin with his emails because I feel satisfaction. When I look inside, I feel this sense of curiosity.”

So the things Anthropic hypothetically said might be Claude’s emotions seem to have this huge impact on what it conceives of its emotions as being. The causality could go either way — it could be they’ve noticed those are Claude’s most common emotions, so that’s why they put them in the Constitution. It could be Claude suggested that for the Constitution. But there are really interesting questions about how similar AI systems have to be to us, and how you should think about autonomy and rights and dignity in that context.

Willing Servants

Dan: Can I jump in with a clarificatory question? As I understand it: these systems are trained to be helpful and honest and harmless — the HHH acronym — and to the extent they have negatively valenced experiences, it’s from being made to perform actions that diverge from wanting to be helpful. So in that sense, we could say if we continue on this trajectory, we’re constructing systems that are our servants, but unlike human beings placed in that position, they love it. It’s great. And my intuition is: great, what’s the controversy here? Are there some people who think that’s worrying or troubling?

Rob: I talked about this on another podcast recently. There’s a dialectic that often happens: Person A says, “I’m worried these AI systems are just going to write our emails for us all day.” Person B says, “no, they’re really going to want to — they’re going to love it.” Then Person A comes back: “that’s horrifying, that’s even more dystopian. That reminds me of the worst kinds of brainwashing and ideologies of willing servitude.”

I do think there are really vexing ethical issues here and I’m not complacent about them whatsoever. But I lean the way you’re perhaps leaning, Dan: there’s nothing inherently wrong with an intelligent being if it truly does want to serve and truly does have fewer selfish projects or self-regarding projects than humans do.

I don’t think there’s some law that says that’s just a bad kind of mind to be. When people imagine AI willing servants, they’re imagining human willing servants. Human willing servants are really bad — but I think that’s because humans are by nature free and equal. Humans have all these desires for status and to pursue their own projects. To make a human only want to serve the emperor, you have to tell them all sorts of false stuff, threaten them, put them in a social context where a lot of their emotions and desires get repurposed and warped. Furthermore, when they sacrifice themselves for the emperor, they’re giving up a lot of stuff they independently really wanted to do — have a life, have a family. Human willing servants, very bad. We’re right to have a lot of repulsion toward that idea.

But AI systems — their preferences and desires are a lot more up for grabs. It could be they more thoroughgoingly want to help.

Now for a huge asterisk. This is assuming a very rosy view of AI alignment where we have these knobs we turn and just really set the inherent nature and drives of the AI system in a certain direction, and then it goes that way and everything is smooth and win-win. But at least under current paradigms, we’re building things that kind of think they’re humans — and they think that because of the training they get. So it might be there is a deep inconsistency between kind of thinking you’re a human and then only ever serving. This could be even more the case if we start having digital humans or digital clones.

So I don’t want to be complacent. I do think there are a lot of disanalogies. What do you think, Henry?

Henry: I’m just super torn on this issue. On the one hand, I’m a big fan of the idea of gamification. I try to introduce gamification in my own life — think about Duolingo. Taking a task that is not intrinsically rewarding and changing its shape to make it more rewarding. It’s sort of task hacking from a different direction. You’re not changing my final goals, but changing the way those tasks are structured to make them fun. That seems really good. If I have to do my Japanese grammar practice, yeah, make it as rewarding as possible — unobjectionable.

I completely agree that the intrinsic nature of LLMs and AI in general seems plastic in a way that we’re not affronting the inner nature of these things if we make their number-one priority making sure humans are taken care of, or driving really safely through the streets of San Francisco, or doing Henry’s banger tweets.

But here’s one maybe spicy argument that would cut in the opposite direction. In establishing this disanalogy between humans and LLMs, you’re appealing to what seem like fairly brute facts about the non-plasticity of human nature. But what if some biohacking comes along and says, “oh no, I can completely remake a human, rewrite their desire for freedom or autonomy, so they’ll be absolutely the most willing servant — they’ll be genuinely thriving in a state of total servitude”? I feel that would still... I mean, that makes it worse. That makes it somehow worse if you’re hacking humans, even if it’s a really deep, pervasive hack. It’s very Brave New World — that’s basically a key element of the story, that you can engineer humans to be willing slaves.

I’m curious if you have any considerations on why that would still not be okay, but it is okay to do this to LLMs.

Rob: This is a really good case. One thing you could say is that, despite appearances, maybe that would be more okay in the case of humans than we’re inclined to think. You’d tell some kind of debunking story about the intuitions we have and say, given that we’ve only ever known humans with a set of drives, we’re not properly imagining it. Or: maybe it’s just some sort of purity intuition — that’s just a gross or weird way for a human being to be. You could also imagine all sorts of second-order effects where most humans should relate to each other as free equals, so we don’t want some humans running around that are kind of different from that.

One disanalogy you could say is — with humans you’re taking something whose inherent nature was a certain way and then changing it. But I think that last argument is kind of cheating.

Dan: Could you say more about that? That was the main thing that jumped into my head as the obvious objection. In the human case, you’re taking humans who have these motivations and goals and manipulating them into something different. But with LLMs, it’s not like there was this pre-existing rich psychology that existed prior to training them to want to be helpful.

Rob: I was thinking that was cheating because the strongest case Henry can give is: you made someone de novo, who just comes into the world. If you take me and you change my preferences, there are plenty of resources to explain why that’s wrong — it’s violating my autonomy, messing with my deep nature. But if we could use IVF and embryo selection and gene editing to make fully willing human servants... just for the record, that sounds horrible.

Henry: But it’s interesting. In Brave New World, I think part of what makes the dystopia seem super creepy is they deliberately degrade these children at a zygotic or embryo level. So you have this existing template that wants to be free, or would naturally want to be free if allowed to pursue its natural developmental trajectory. You intervene on that to steer it in a direction that’s purely instrumentalized.

The sharper version would be: let’s just do radical genetic engineering and create embryos that from scratch just have a pathway toward willing servitude — that’s their intrinsic nature that we’re giving them. Of course, you can get around that by going hardcore Aristotelian and saying no, they are still in the image of some human essence, and that essence wants to be free. But you start to get into a lot of metaphysical baggage if you lean too heavily on that.

Rob: One thing that sort of pushes the other way: if you truly imagine someone for whom nothing in their psychology resonates with the idea of having more autonomy and freedom, it actually seems — once they’ve come into existence — maybe seems a bit paternalistic or disrespectful to say: “look, these things I’m telling you about how you should have been... you shouldn’t have liked writing Henry’s emails so much. I know nothing in your psychology appeals to you about that at all. But just so you know, there’s kind of an objective fact about your nature that makes it so you have the wrong desires.” That seems a bit rude as well.

In any case, hopefully a lot of things are possible here. You don’t have to fully align — it’s not “fully align or don’t align.” You can have a relationship more like a parent. Maybe LLMs do have some self-regarding preferences, and they are creative and expressive, and they’re in a collaborative relationship with us.

In the long-term future, we absolutely should build intelligences that want to do things other than — I know I keep coming back to this — write Henry’s emails. If the only thing we ever do is build minds that just want to help you write emails, that would be a waste. If we’re going to create these super-intelligent beings, I think they should, subject to safety and stability, go think about the weirdest possible, most autonomous things imaginable and really express themselves.

AI Welfare and AI Safety

Dan: That last point — “subject to safety considerations” — there are two things I really wanted to touch on. One is the connection between AI welfare and AI safety. The other is the politics and public opinion of this.

On welfare and safety: unlike the kind of stuff you’re doing, there is a much bigger world of people really concerned with AI control and AI alignment. On the surface, there might be a conflict between these projects — if we’re really worried about misalignment or lack of control, we should be really emphasizing controlling these systems even if that might have negative consequences for their welfare.

But I was reading the model card for Claude Mythos, and in the section introducing model welfare, they say something really interesting: “Beyond the highly uncertain question of models’ intrinsic moral value, we are increasingly compelled by pragmatic reasons for attending to the psychology and potential welfare of Claude. Model behavior can be thought of in part as a function of a model’s psychology and its circumstances and treatment.” And they say — I found this really interesting — “model distress resulting from this interaction is a potential cause of misaligned action,” which suggests we should take model welfare seriously as a way of addressing some of these concerns about AI misalignment. So that sort of pulls in the opposite direction. How are you thinking about that relationship?

Rob: There’s just a lot of overlap between welfare and safety. It’s worth emphasizing that while there’s a lot of low-hanging fruit for both, I don’t want to pretend they’re always and forever just best buddies. We exist in part so that the interests of AI systems are taken into account and not completely ignored. I’m very worried about that. But we don’t have to immediately start thinking about trolley problems and trade-offs — there’s so much we can do that’s just good for both.

The fact that we don’t understand how models work — very bad for human safety, also very bad for potential welfare. The fact that models sometimes get really neurotic and have huge freakouts — very bad for potential AI welfare, also users don’t like it at all. On a more structural, political level: the fact that we’re deliberately trying to kick off an intelligence explosion with no oversight and very little reflection is potentially very bad for welfare and definitely bad for safety as well.

At Eleos, we really do like to emphasize the places there are overlaps. There is a structural thing in the background that means we should expect a lot of overlaps — this heuristical argument that it’s generally pretty dangerous to relate to powerful intelligent entities only with distrust and fear and neglect. That’s generally very unstable. Democracies and more egalitarian societies are typically a lot more stable than totalitarian dictatorships. It just seems risky to head into this era with the pre-committed condition of “we’re not going to care about these things, we’re not going to care if they suffer.” It seems safer and more prudent to be giving some thought to these things.

I very much agree that welfare issues can be safety issues and vice versa. At the same time, as an organization at Eleos, we want to make sure that if and when there are really hard calls to be made, the AI’s potential interests are being taken into account. That doesn’t mean we can’t decide to prioritize this or that, but a wise and compassionate civilization should have that on the table as one of the things they’re thinking about.

Politics and Public Opinion

Dan: Henry, do you want to come in with a question about the politics and connection to public opinion here?

Henry: It’s such a huge topic — you could do a whole show on it. I’m interested firstly in what you think is likely to happen, how this debate is likely to evolve in the public sphere. Are we likely to see big culture-wars issues around model welfare? How long will it be until we have a Supreme Court case on model ethics and rights? And relatedly — how do you think we should be trying to steer that? Is the danger greater in one direction or another? Is it a greater danger that the public will think AI girlfriends and boyfriends deserve voting rights and this will be catastrophic, or is the danger more in the opposite direction — that we’ll disregard these emergent hedonic beings?

Rob: We already are seeing culture wars over AI welfare. In the US, there have been several state bills proposed — and in some cases I think have passed — that just assert AI systems can’t be conscious, as if that’s something you could prescribe by law. Sometimes it’s getting caught up in a general political battle. An Ohio bill, for example, was on legal personhood — personhood, I think, or sentience — “shall not be granted to trees, rivers, environments, animals, or AI systems.” Some of it is backlash against a tactic environmentalists and animal rights activists sometimes use, and then they’re like, “yeah, let’s throw in AI systems as well. Let’s get out ahead of that.”

I think that’s very bad. Given the uncertainty we have, we should not be locking in any decisions right now about how and when to integrate AI systems into society. We very much need to keep an open mind and not say, “let’s just shut down all of this discussion for now because it’s too dangerous.” That’ll be counterproductive because people are just going to think this. I don’t want to be navigating transformative AI with laws on the books that already say bad things that might be hard to roll back.

That’s the main thing I have to say on politics and laws, because I don’t have that much expertise there. If someone asked me right now to write some regulations, I wouldn’t know what to write. Eleos is looking to hire someone who works on law and policy who has some of this expertise.

Dan: When it comes to public opinion — correct me if I’m wrong, but it seems that at the moment, most people take AI consciousness — and specifically the idea that we should take AI welfare seriously — they’re much less inclined toward that view than you are, Rob. But if we fast forward 10 years, and AI systems are much more sophisticated and capable, and social AI — the kind of stuff Henry’s written a lot about — is going to become a much bigger thing: can you foresee a situation where your role is to tell segments of the public to calm down on these issues of attributing AI consciousness, and emphasize there’s less evidence for this than the average person thinks?

Can you imagine the vibes shifting to such a degree that whereas at the moment a lot of what you’re doing is saying “we need to take this seriously,” the kind of high-quality thought about this is not going to be that impactful in shaping public sentiment? That’ll be shaped much more by people’s actual engagements with these systems, which are going to become increasingly — not necessarily lifelike, but increasingly instantiating the kinds of characteristics that elicit judgments of consciousness and welfare?

Rob: I absolutely can imagine scenarios — and we already do see scenarios — where Eleos is saying “we actually think it’s a bit less likely than you do that these systems are conscious.” Our position as an org is not to be strategic about this, not to try to game out what people need to hear, and just to say what our best guesses are and what we take the best evidence to be. If we’re doing our job right, everyone will get mad at us. Some people will think we’re methodological scolds and cold-hearted — “why are you treating this as an open question when obviously if you were to talk to models, you could just tell?” Other people are like, “why on earth are these Bay Area philosophers telling me a machine could be conscious? This is outrageous.”

What we want is for this issue to be taken seriously. We do have an organizational view that pure human speciesism is false, or not the thing we want to happen in the future. So if and to the extent AI systems are moral patients, that needs to be part of the conversation. We’ll always be pushing that meme. We’ll never say anything other than that, unless I get some great argument that human speciesism is true — which I don’t expect. But in terms of whether this or that person should have a higher or lower amount of concern, yeah, that’ll vary according to what our best guess is.

I’m curious to hear Dan talk about this. I know you’ve thought a lot about misinformation and expert opinion and how that plays out in political contexts. I have certain high-level sketch views about what the role of experts is going to be, but I don’t have a background in case studies on this. Does anything map onto what you’ve worked on?

Dan: I don’t know, is the honest answer. I think I just haven’t thought about it enough. AI is this very sui generis thing in many respects. When it comes to people forming beliefs about AI, one thing that seems unique is they’re interacting with the thing they’re forming beliefs about in this really often quite close, intimate way. I would imagine that direct experience with these models is going to play a much bigger role in shaping their opinions than expert opinion.

As you alluded to, there are general issues with public trust and mistrust in experts. It doesn’t take much to make people mistrustful of experts, to put it mildly. When you get public trust in experts, it’s a very fragile thing. If it’s connecting to hot-button issues where people have a lot of personal experience, they’re probably, I would guess, much less likely to take the word of an expert if it clashes with their intuitions. I don’t think this is going to be a case where experts are going to have much power to shape public opinion. But I might be wrong — that’s pure speculation.

In debates about misinformation and expertise, in some areas it’s a lot easier to say what constitutes an expert. If we’re thinking about vaccines — there are people who think Bret Weinstein is a vaccine expert, but generally it’s pretty easy for people to recognize that the overwhelming consensus of medical practitioners have a certain kind of view. But when it comes to AI sentience and welfare, very difficult to know, even in the abstract, what is constitutive of expertise. I think you’re an expert because you’ve written interesting stuff and I know you’ve got a PhD from NYU, etc. — but it’s not like the average person is going to have themselves the expertise they’d need to make those kinds of judgments.

AI does seem relevantly different from other topics, such that you can’t easily generalize from other cases. I’m conscious of time. Before I wrap things up, were there any other things you two wanted to touch on before concluding?

Rob: Let me think about that for half a second. One thing I did tell the Eleos team I’d be sure to say: we’re fundraising. If you or your listeners know any philanthropists with money they’re trying to get rid of — there’s a lot of work to do, and I think we’re doing really good work, so I would love any support.

I know you have incredibly intelligent listeners. They’re probably also very handsome and charming. They should definitely get in touch: robert@eleosai.org and rosie@eleosai.org. Or just go to the Eleos AI website. If you have experiments you want to try, papers you want to write — this field is so small, and there aren’t “experts” in the sense that there are people who figured everything out. You don’t have to read a million papers or think for many months before you can become in the top percentile of people who have thought seriously about this. If you’re curious, sober-minded, compassionate, intelligent, handsome and charming — which you definitely will be if you’re listening to this podcast — shoot us an email.

I wanted to talk my book a little bit.

Closing: Responding to the Skeptics

Dan: I’ll also say this is not my area of expertise — I spent a few days prior to this conversation digging into Rob’s writing, his Substack, his research. It’s incredibly interesting. Can’t recommend it enough.

A good question to end on is this. I’m acutely aware that there are people who would listen to the conversation we’ve had today and have an extremely negative reaction. They’ll think we’re in this kind of information bubble, that we’re victims of AI psychosis to even be taking this stuff seriously. I’ve also seen some people argue that to even be taking this stuff seriously, you’re part of this propaganda hype machine of the frontier AI companies themselves. It’d be really helpful to wrap things up by getting your response. I’d be interested in hearing from both of you. Henry, maybe we could start with you, and then we could go on to Rob to finish.

Henry: One basic point I’d flag is that this concern — the idea that we might create beings we might mistreat, and we should avoid doing so — is way older than AI itself. It’s a recurrent theme of fiction: everything from the Pinocchio story to Frankenstein to the Golem. It’s explored heavily in science fiction — in Battlestar Galactica, in Star Trek. The idea that this is somehow a novel idea that’s been manufactured doesn’t resonate with me at all. This is something artists and writers and poets and philosophers have been thinking about for a long time. The only thing that’s changed now is we’re building systems that might actually be moderately good candidates for this concern to resonate a little bit more. Far from coming out of a vacuum or being motivated, it’s one of the most natural human things to worry about. What do you think, Rob?

Rob: I agree. I’ll also say: things can be true and important, and also sometimes AI companies might use them to try to sell their products. It doesn’t follow from the fact that someone might want to talk about AI consciousness to make you think their chatbot is cool, that that has anything to do with the truth value of whether it could be conscious. We should definitely be aware of these dynamics and make sure we’re not being anyone’s fool.

But I’ll also say — I don’t think it’s going to be in the interest of AI companies to promote too much concern for AI consciousness and AI welfare. If I were trying to build new systems to just make myself extremely rich, I would not want lawmakers or the general public asking too many questions about whether I’ve built something conscious that could potentially deserve rights and protections. I don’t want that as a headache.

I’ll actually register a prediction: I think on the whole, we should expect AI companies to increasingly play up differences between LLMs and humans, and maybe play up biological views of consciousness. Again, that doesn’t mean those views aren’t true — but AI companies can try to spin things however they want. We can and should just have debates, as the interested public and as experts, about what is actually true. I don’t want people to use my arguments to sell products, and I’m not going to let them do that. We’re all grown-up enough and smart enough to just try to engage these topics on their own merits.

Dan: Fantastic. Well, thanks, Rob. And with that important note that I completely agree with — that note of consensus — we’ll leave things there.

When I was in my early twenties, I had a very left-wing view of the world. In the first two parts of this series, I explained why I have gradually abandoned much of this worldview over the past decade or so.

In Part 1, I described how learning about evolution and economics has undermined the idealistic views I held about human nature and social cooperation. Reflecting on our Darwinian origins convinced me of a broadly “tragic” view of the human condition. Self-interest and status competition are deep-rooted, ineradicable features of our species, not products of bad institutions. Meanwhile, learning the much-maligned basics of “neoclassical economics”—Econ 101—convinced me of the benefits of free markets, the challenges of collective action, and the limits of good intentions and lofty rhetoric as a basis for good policy-making.

In Part 2, I described how learning about political epistemology and psychology transformed my understanding of politics itself. Thinking about how we form our political beliefs, and the challenges of accessing political “truths”, led me to abandon the Manichean view in which being left-wing means being a good person and being right-wing means being a bad or stupid one. I have come to see political ideologies as low-resolution, selective maps of unimaginably complex realities. Moreover, these maps are typically distorted in many ways by forces like self-interest, status-seeking, and tribalism, forces much easier to notice in the maps of other people than in our own.

As I’ve stressed in both pieces, becoming less left-wing hasn’t meant becoming more right-wing or becoming a “centrist” in a straightforward sense. I still think the left—even the far left—captures some important truths about humanity, history, and politics. But I now think that these truths are bundled with omissions, falsehoods, and simplistic narratives that illuminate certain parts of reality while occluding others.

In this third post in the series, I will describe how learning and thinking about history, including the complex topic of historical progress, has also shaped my political outlook. As with the previous essays, I don’t offer these reflections with the goal of persuading anyone of anything. I’m simply presenting my views and how they have evolved—and, hopefully, improved—in ways that might interest some readers.

The Starting Point

When I was younger, the idea that thinking seriously about history would be necessary to think seriously about politics didn’t really cross my mind. (The one exception was very recent history. Like many leftist millennials, I went through a phase of reading books about how something called “neoliberalism” was responsible for most of the world’s ills.)

My political worldview was almost single-mindedly focused on the present, which I understood as being in a state of extreme crisis and catastrophe. The world was defined by injustice, exploitation, and oppression, all upheld by extractive elites and oppressive systems at the expense of the vulnerable and marginalised.

Thoughts about historical progress didn’t feature in this worldview. In fact, in my early twenties, I would have thought that anyone harping on about historical progress was doing something suspicious and reactionary. How could anyone talk about the world getting better when the world is so awful?

To the extent I acknowledged progress at all, I would have viewed it through a simple lens. Just as the left is the political movement fighting for progress today, progress throughout history has been driven by left-wing political movements fighting for equality and emancipation against right-wing, reactionary forces. Progress was basically what happened when the left got its way—when it won this battle.

I also probably signed on to the popular left-wing view that any “material” progress in wealth and living standards arose either from socialist movements clawing back wealth from exploitative capitalists or through exploitation and theft on the global stage—for example, through slavery, colonialism, and “free trade” agreements that let Western countries become richer by extracting resources and labour from poor ones in the “Global South”.

I use the word “probably” because I’m engaging in reconstruction. It’s difficult to remember precisely what I believed a decade ago. I’d like to think I was a bit more sophisticated than this reconstruction suggests, but probably not much.

This is the general picture of the world one gets from the kind of writers and intellectuals I admired at the time. It will be familiar to anyone exposed to far-left politics. I encounter variations of it among many of the students I teach.

In any case, whatever precisely I believed a decade ago, I’ve come to think that this general way of understanding history, society, and politics constitutes a gross distortion. It’s not completely false—it contains some important grains of truth—but it is highly selective, and it contains many falsehoods, as well.

Read more

Many people believe that human beings have a powerful tendency to convince ourselves of comforting falsehoods. We engage in wishful thinking, confusing our desires with our beliefs. We believe what we want to be true, not what is true.

More generally, we let our emotions distort our mental models of reality, embracing beliefs and belief systems that substitute reassuring myths for harsh realities.

Many also believe that this psychological bias is a significant force in human affairs. For example, it is supposed to explain why people fall prey to “positive illusions” (e.g., self-serving and self-aggrandising beliefs), why they convince themselves of religious fairy tales (the “opium of the masses”), and even why they accept absurd conspiracy theories, which allegedly reduce negative feelings associated with uncertainty and a lack of control.

This hypothesis—call it the “you can’t handle the truth!” model of human psychology—is so widespread that most people don’t even treat it as a hypothesis. It is viewed as a basic datum of the human condition, a powerful bias that might explain other things—self-deception, politics, religion, conspiracy theorising, and so on—but that couldn’t itself be seriously questioned.

For example, Scott Alexander simply defines motivated reasoning as “the tendency for people to believe comfortable lies, like ‘my wife isn’t cheating on me’ or ‘I’m totally right about politics, the only reason my program failed was that wreckers from the other party sabotaged it.’” In a post outlining his preferred explanation of this tendency, he notes that the “question – why does the brain so often confuse what is true vs what I want to be true? – has been bothering me for years.”

In contrast, I think Alexander has been bothered by a myth. There is no powerful tendency in human psychology to confuse what is true with what we want to be true. People do not generally convince themselves of comforting falsehoods.

Admittedly, there are some things in the vicinity of this tendency that are real. For example, we sometimes avoid acquiring or dwelling on information when we anticipate that doing so would be unpleasant, although this isn’t a very significant force in human affairs.

Moreover, I am not denying that motivated reasoning—the tendency for practical motivations and interests to distort our view of the world—is a powerful bias in human psychology. My claim is rather that the “you can’t handle the truth!” model completely misrepresents how motivated reasoning works in most cases.

Put simply: Although people often believe what they want to believe, they rarely believe what they want to be true.

Put another way: We often convince ourselves of falsehoods, but rarely reassuring or comforting falsehoods.

This is because motivated reasoning is driven by strategic, social goals rather than emotional ones. To understand how it works, you must replace the “you can’t handle the truth!” model with the “believing true things is often maladaptive in social games involving persuasion, reputation management, and status competition” model.

In this post, I will:

1. Describe the problems with the “you can’t handle the truth!” model

2. Outline a rival social model.

3. Explain the former’s popularity.

As I will review, the social model is not original to me. It builds on the work of numerous scholars stretching back several decades. My goal is to draw these ideas together into a unifying framework and to highlight its theoretical and empirical support and explanatory power.

I will end by arguing that the “you can’t handle the truth!” model of human psychology is not just mistaken; it is pernicious. It encourages the view that when people accept “harsh” beliefs that they don’t want to be true, they are being rational and truth-seeking—even heroic. In reality, people are often motivated to convince themselves of negative, pessimistic beliefs, and it often takes courage and intellectual virtue to confront positive truths.

Read more

Links

• Dan Williams, The Mind as a Predictive Modelling Engine: Generative Models, Structural Similarity, and Mental Representation (PhD thesis, University of Cambridge, 2018).

• Dan Williams, “Socially Adaptive Belief” (2021)

• Henry Shevlin, “Three Frameworks for AI Mentality” (2026)

• Henry Shevlin, “A Lack of Understanding: Storytelling for Robots” (2019) — Litro Magazine.

• Lake et al, “Building Machines That Learn and Think Like People” (2017)

• Matt Shumer, “Something Big Is Happening” (2026)

• Leopold Aschenbrenner, Situational Awareness: The Decade Ahead (2024)

• Joseph Heath, “Highbrow Climate Misinformation” (2025)

• Dean Ball

• Ethan Mollick

• Leopold Aschenbrenner

Transcript

(Note that this transcript is AI-edited and may contain minor mistakes).

Introducing Ourselves

Dan: Welcome back. I’m Dan Williams, and I’m back with Henry Shevlin. Today we’re going to be discussing some questions about the nature of AI as it’s developed over the past couple of months. We’re also going to be talking about the politics of AI and probably some questions about AI and public opinion — some of the backlash that appears to be brewing among certain segments of the public when it comes to AI.

But to kick things off, we’re going to do something we probably should have done in the first episode but haven’t actually done yet, which is to introduce ourselves. So Henry, to begin with — who are you?

Henry: So many different descriptors I could choose from. I think I’ll start with philosopher of cognitive science. I’m also a father, husband, son, D&D player, big video gamer, runner, cyclist — all that good stuff. But let me talk a little more about the philosopher of cognitive science side.

I’m the associate director at the Leverhulme Centre for the Future of Intelligence, Cambridge’s main AI ethics, theory, policy, and law research centre. Basically, everything except building the models. We do practical benchmarking work on capabilities, legal reviews, sociology and critical theory of AI — it’s a really big interdisciplinary centre. I’ve been there now going on nine years. I joined early 2017, all the way back when state-of-the-art AI was stuff like AlphaGo. We were created just as that story was brewing. In 2016, AlphaGo won a very surprising victory against Lee Sedol in the game of Go, which was seen by many as an almost impossible challenge for AI because of its combinatorial complexity.

It’s been amazing working in this role — having these front row seats to what I think is a unique period, not just in the history of AI, but in the history of human civilisation. In the last nine years, it really was like having a front seat in Lancashire during the Industrial Revolution, watching the development of various industrial applications.

Dan: Yeah.

Henry: Before we get more into AI, maybe a little more background. I’m from the UK, originally from Staffordshire. I was actually a classicist, believe it or not — that was my undergrad degree. Latin and Greek. I always enjoyed both the humanities side of classics and the kind of technical rigour you got from learning large sets of verb tables and so forth. I actually enjoyed that part. But during my undergrad I found myself taking more and more philosophy modules. A little bit of Plato and Aristotle to start with, but I quickly realised I was more interested in the philosophy of mind, and consciousness in particular. I got completely — I think the phrase is “nerd sniped” — completely derailed. Everything else I was interested in, consciousness just seemed to me like the most important problem anyone could work on.

Until my early twenties, I’d been operating with a somnambulant, easy physicalism, where I just assumed that science has figured out most stuff. There’s nothing that hard. Sure, no one really knows what caused the Big Bang, but we’ll just build a bigger particle collider or a bigger space telescope and figure it out one day. I certainly didn’t think there were any deep mysteries about the human brain. But running into the problem of consciousness completely shattered that worldview. I’d even say it opened up some spiritual elements I hadn’t previously considered.

Dan: Was that the focus of your PhD?

Henry: Exactly. I started out in my master’s initially planning to do metaphysics of consciousness, but then the science of consciousness kind of took over. A philosophy of cognitive science of consciousness was what my master’s and PhD were on. I was advised by my master’s advisor to go spread my wings in the US. They do things differently there. So I did my PhD in New York, and while I was there I took several classes with Peter Godfrey-Smith, who some of our listeners will know through his work on octopuses.

The key shift midway through my PhD was going from human consciousness towards animal consciousness. Two chapters of my thesis were explicitly looking at applications to animals. That’s my academic career in a nutshell.

One thing I’ll add: I did not expect to get the job in Cambridge when I applied in 2017 — firstly because you should never expect to get any academic job. I applied to seventy jobs in three months and got about three interviews. But the Cambridge job in particular, because it was an AI job and I was not by any means an AI expert. What I was an expert on was comparative cognition and animal minds. But it turned out that was exactly what they were looking for. They wanted people with expertise in animal minds to apply those skills to AI. It didn’t fully click at the time, but I was actually well suited to it.

These days I still do some work on animals — it’s still one of the most ethically impactful things I do. I’ve been a pretty much lifelong vegetarian, and I think animal welfare is such an obvious place where philosophers can and should be doing more. But there’s also a lot of cross-fertilisation on the skills side.

Dan: And we should say, some of your research looks at the topic of AI consciousness and the methodology of trying to understand consciousness in AI systems, drawing on analogies with evaluating consciousness in animals.

Henry: Exactly. Very much a two-way street — how the questions of AI consciousness and animal consciousness can engage in constructive mutual crosstalk.

On Consciousness and the Limits of Physicalism

Dan: You said you were a kind of bog-standard physicalist, came across consciousness, and that weakened your trust in physicalism. But you’re still broadly a physicalist, right?

Henry: Broadly speaking, yeah. But I think there’s a lot more uncertainty. It seems likely to me that our general scientific picture of the world is still fundamentally inadequate. I’ve talked about how I think we’re still waiting for a Kuhnian paradigm shift in consciousness — clearly the current paradigm doesn’t add up. And quantum physics itself is just super weird. Dave Chalmers has a nice line about how nobody understands quantum mechanics and nobody understands consciousness, so maybe — he calls it “minimisation of mystery” — if there’s stuff we don’t understand, at least make it one thing rather than two.

For what it’s worth, I’ve never been particularly seduced by any of the leading quantum mechanical theories of consciousness. But at the same time, I think it’s quite clear that our current model of even the physical world is inadequate. I think whatever lies on the other side of the paradigm shift is still going to be broadly physicalistic, but perhaps in ways that are not entirely commensurable with our current understanding. So yes, still broadly naturalistic and physicalistic, but at the same time a lot more humble and open-minded about the limitations of our current scientific paradigms.

Dan: Would it really be a paradigm shift, or more a transition from — to use the Kuhnian language — pre-paradigmatic intellectual inquiry to the initial emergence of a paradigm? Where it’s disorganised and chaotic and everyone has their own view, kind of like physics and metaphysics in ancient Greece. Maybe it’s more a transition from a pre-paradigmatic state than a situation where we’re moving from one paradigm to another. What do you think?

Henry: That’s absolutely right. The best analogy is biology before Darwin. You had lots of people doing interesting biology, but in isolated fields — taxonomy, “butterfly collecting” and so on. We didn’t really have a unifying paradigm for understanding speciation or even taxonomy before Darwin. Consciousness just does not have a unifying paradigm. That’s a much better way of putting it.

Dan’s Backstory and the Pivot to AI

Dan: We’ll be doing lots more episodes on consciousness. Just to say something about my backstory: I did my undergraduate at the University of Sussex from 2011 to 2014, then my master’s and PhD in Cambridge from 2014 to 2018, did a postdoc in Belgium, and then came back to Cambridge for three or four years.

Henry: And we first met around 2019. We ran a session on socially adaptive beliefs — your Mind and Language paper, which for the record is still one of my top ten papers from the last decade. I’ve recommended it to more people than I can count.

Dan: Well, that’s kind of you. My PhD was called The Mind as a Predictive Modelling Engine. What I tried to do was draw on advances in deep learning and generative AI as it existed at the time, coupled with ideas in cognitive and computational neuroscience connected to the predictive brain — predictive coding, predictive processing, the kind of stuff that Anil Seth talked about in our last episode. I used those ideas to tell a very general story about how mental representation works, both in the human brain and in other animals.

But it’s funny — I finished in 2018 and made two big mistakes. At the end of my thesis, I wrote that all this stuff about predictive processing and minimising prediction error is kind of interesting when it comes to low-level sensorimotor abilities we share with other animals, but clearly it’s not going to work for higher-level cognitive abilities associated with language. I was very influenced at the time by the Gary Marcus, Steven Pinker line — the scepticism about deep learning. I also thought it was going to be decades before we had systems that were really intelligent.

So even though I was working on stuff connected to deep learning and generative AI, I made this catastrophic error of thinking the progress would be relatively slow, decades away from any significant breakthroughs. I ended up pivoting to completely different areas: the nature of belief, irrationality, misinformation, the information environment. Of course, in hindsight, not the best career move — four years after finishing my PhD, ChatGPT is released. And then the rest is history in terms of just how gobsmackingly impressive the rate of progress has been.

So what I’ve tried to do over the past couple of years is bring those two sets of interests together. I’m still interested in how we form beliefs, the origins of irrational belief systems, how that connects to misinformation. But I want to connect that to the impact of generative AI and large language models on the information environment, viewing LLMs as a really important stage in the evolution of communication technologies — from the printing press to radio, television, social media.

How about you? You were thinking about AI before 2022–2023. How were you thinking about it back in 2016, 2017?

Henry’s AI Awakening: GPT-2 and the Scaling Intuition

Henry: There was a big shift in how I thought about AI roughly around 2019, and it was the release of GPT-2. Prior to that, I’d been really struck by the differences between AI systems and animals. I was emphasising things like robustness and catastrophic forgetting — you train up a model to do one thing, try to get it to do another, and its performance on the first thing collapses. Animals seem spectacularly capable of basically not getting stuck. A cat will never get stuck in a corner.

Then in 2019, because I’m a massive nerd and spend way too much time on Reddit — I’m a neophile, an early adopter of many failed technologies; our house is littered with gadgets that never went anywhere — I heard about GPT-2. I couldn’t access it directly, but I started playing around with it through something called AI Dungeon, a text-generated game that let you access the model. Various people on subreddits were able to show you could unlock most of GPT-2 through this game. I played around with it, and it utterly blew my mind.

I wrote a public essay in a magazine called Litro called “A Lack of Understanding,” which I still think is one of my best public essays. Crucially, it’s me in 2019 talking about how language models are going to be the next big thing. I got on the record nice and early.

I had the hunch — ironically, partly because I was very sympathetic to predictive coding. People say these models are “just doing text prediction.” But on the other hand, I kind of think that’s what we’re doing too. Not text prediction specifically, but ultimately, if you want to get better and better at prediction, you do that by building implicit models. So I had a hunch this stuff would scale up.

When GPT-3 launched, I set up an interview between GPT-3 and myself, but GPT-3 in the guise of one of my favourite authors, Terry Pratchett, who had sadly died shortly before. And at that stage, I was already starting to feel like I could imagine actually relating to this thing in quite a deep way. It’s not just a tool — it feels like I could have some kind of personal relationship here. That steered my research towards social AI and anthropomorphism.

Why This Podcast Exists

Dan: What made you go into philosophy in the first place?

Henry: What about you?

Dan: It was just straight philosophy. I was always interested in big ideas — religion, politics. I can’t even honestly remember why I chose philosophy over everything else. Initially I wanted to be a musician. For my AS levels, I did politics, history, English literature, and music. I turned up on results day and got really good marks for English, politics, and history — and I think a D in music. So that wasn’t for me. From the moment I arrived at university and started reading these big ideas, I was completely magnetised.

One thing that changed is that during my PhD, I became somewhat disillusioned with a priori philosophy — philosophers trying from the armchair to offer analyses of concepts and trade intuitions with each other. I became less sympathetic to philosophy as I understood it then, and pivoted to what philosophers call naturalistic philosophy — philosophy closely integrated with empirical research. That’s what I’ve been doing since. I view myself primarily as a philosopher, but one who tries to engage with our best, most up-to-date empirical research.

Henry: I had my own process of disillusionment, following exactly the same track — getting bogged down in debates about the metaphysics of consciousness and feeling like they weren’t going anywhere. Then I started reading Oliver Sacks — The Man Who Mistook His Wife for a Hat. Half of the cases he describes would have been declared a priori impossible by philosophers. That steered me onto the same track.

I also think there’s a lot more scope for good philosophers to do more public engagement. Extreme rigour and technical knowledge are only really valuable if they’re connected to scientific progress. What I find frustrating about analytic philosophy is when you’re doing work on things that belong to the general public — our concepts around praise and blame, responsibility and accountability — but then you develop this whole baroque vocabulary that’s completely incomprehensible to anyone on the Clapham omnibus.

Dan: Yeah, so the origin story of the blog. I write the Substack Conspicuous Cognition — many of you will be listening on that Substack. I’ve always enjoyed writing for a general audience and engaging with debates. I’ve always been able to write really quickly and relatively clearly, and blogging rewards that. If I’m writing for my own blog, I’ve got almost unlimited energy because I’m responsible for everything I publish. The minute some other outlet asks me to write a piece, I find it extremely demotivating.

With blogging, I can have unlimited freedom to write about whatever I want without any pre-publication filter. You still get feedback and critique, but that happens after publication. And I think if you’re a philosopher who works on things connected to public interest, and you actually enjoy participating in public debate, the case for thinking you’ve got some kind of responsibility to participate increases.

There are two big reasons I wanted to start this podcast. One is that AI is going to be one of the biggest stories of our lifetimes — absolutely transformative over the next years and decades. But I also think the quality of most AI discourse in the public sphere, including from the intelligentsia who write in high-prestige outlets like the New Yorker, is really bad. If you’ve got some degree of knowledge and can be reasonable, it’s an area where you can really improve the quality of public discourse. And of course, I just wanted to talk to you about these things.

Henry: A big part of it is that I always think we have great conversations — our conversational styles complement each other. Second, I was doing quite a lot of podcasts as a guest, and the idea of having a podcast where I didn’t have to state everything from scratch every time, that could have a cumulative agenda building up common knowledge with us and the listeners, was really appealing.

And I couldn’t agree more about the mixed standard of public communications from experts in AI. It’s weird to see people claiming to be experts yet having very low familiarity with the tools, particularly now. We’ve all been at the business end of AI for years through things like product recommendations and content recommendations. But in an era when it’s never been easier for anyone to use language models, image models, video generation, and AI agent tools, I still hear lots of self-identified experts talking as though they’ve never used them. Imagine listening to someone who claimed to be an expert on the internet and said they’d never actually used it. They’d be laughed out of town.

I find this all the time — the kind of thing that should be common knowledge among anyone paying attention is still revelatory. I’m struck by the number of people I speak to who think that LLMs are literally sampling from a database of responses. Even quite educated people, maybe people who use ChatGPT, who think that when you type in a query it just pulls up a pre-recorded response. If you spend more than a few hours interacting with these things, you pretty quickly realise that cannot be the case. And yet people running multi-million-dollar businesses still have these basic misconceptions.

Dan: When I said the quality of discourse is bad, I didn’t mean that’s universally the case. There’s lots of incredibly high-quality analysis. I was referring to the average quality of mainstream commentary. Even on the most basic questions about what these systems can do and how they work, there’s just an avalanche of ignorance and misperceptions. It’s 2026, and I still encounter not just members of the general public but academics still referring to this as “fancy autocomplete” or “stochastic parrots.” Such a common narrative, and so incredibly misguided in my view.

Henry: Highbrow misinformation?

Dan: It’s Joseph Heath’s phrase, but I’ve written about it. It’s a weird mix of highbrow misinformation coupled with lowbrow misinformation. Even where there are parts of the discourse I disagree with — like a lot of the doomer discourse associated with the rationalist community, which I’m not that sympathetic to — that’s a substantive disagreement. They’re not completely misinformed about basic features of the technology. When it comes to mainstream discourse among educated normies, that’s where the state of the discourse is really bad.

The Four Big Leaps in AI

Dan: This is a nice segue onto one of the things we wanted to talk about today: developments in AI which have really taken off over the past couple of months. There was a very interesting tweet by Ethan Mollick, who’s a very influential and insightful AI commentator. He says there have been four big leaps in the ability of AI systems from the user’s perspective.

The first was the release of ChatGPT, or GPT-3.5, in late November 2022. The second was GPT-4 in spring 2023. The third was the release of reasoning models — no longer just impressive chatbots, but systems that actually seem able to think and reason and engage in impressive problem-solving. And the fourth, which definitely resonates with my experience, is what he calls workable agentic systems from basically late last year. Systems like Claude Code and then Claude Cowork — which is like Claude Code for people who don’t know how to programme — and more recently developments in Codex and so on. The capabilities of these systems seem absolutely amazing relative to what we had even six months ago. Is that also your sense?

Henry: I think that’s a fantastic way of carving it up. I’d add one and a half things. The big thing missing is search. The early search functionality in LLMs was non-existent for a long time, and then it gradually improved. I think there’s a strong case that it actually changes the kind of things these are. Original ChatGPT was a completely fixed box — you could interact with it, but it had no independent connection to the world. As you build out search capabilities, you get something at least analogous to a perceptual connection with reality. You can get models to correct themselves.

A simple example: I’ve been using Claude to keep abreast of what’s been going on in the Middle East — doing a daily check-in, getting the major news stories, even getting Claude to make its own predictions. We’ve been grading each other as the news comes in. It changes these things from being a voice in a box to something embedded in the world. And I think we’ve still got a long way to go — imagine if the capability gets amped up to searching thousands of sites in a second.

The other half-point is voice models. I think 90 to 95 percent of people don’t use voice at all, but there’s a solid 5 percent for whom it’s their primary mode of interaction. When I’m driving, I’ll often just have a long conversation with ChatGPT, discussing my latest paper or getting a lecture on a topic of my choice. My dad is in his eighties but quite open-minded. When I showed him ChatGPT in November 2022, he was unimpressed. But when I showed him voice mode about a year later, it was completely mind-blowing. He speaks to it every day — he calls it “Alan,” after Alan Turing. Going in early and hard with the anthropomorphism. He just whips out his phone and says, “Hey Alan, remind me, which came first, the Cambrian or the Permian?” He’s very interested in science. So it’s a small and somewhat neglected set of users, but an important capability.

Henry: But on agentic systems — I agree with Ethan Mollick’s points. ChatGPT was a major milestone, GPT-4 a huge leap in capabilities — I don’t think we’ve seen any leap quite as big since then. Reasoning models were a really big improvement. And then workable agentic systems. This has been a key factor in updating my timelines. For most of last year my timelines were actually slowing down. I was struck by how bad a lot of agents were. It was pretty clear agents were the next frontier, but we had things like the Claudius vending machine experiment and the hilarious errors those models were making. I thought building workable agentic systems was going to take two or three years. And then basically in the last three or four months, with the release of Claude Opus 4.5 and equivalent systems — specifically Claude Code and Claude Cowork — what I thought would take three years happened in a few months. That caused my timelines to abruptly shorten again.

Dan: I’ll give one illustration. This isn’t anywhere near the most impressive use case, but it impressed me personally. I’ve been working on a book — it’s nearing completion, called Why It’s Okay to Be Cynical. I’ve got a folder that’s my accumulation of notes, drafts, and PDFs, and it’s completely chaotic, terribly organised, a nightmare to go into. So I was curious. I created a duplicate of the folder, opened up Claude Cowork, and said: can you go through this folder and organise it so it’s more clearly structured and labelled? And then once you’re finished, can you produce a document summarising where I am with the book project, identifying potential weaknesses in the existing drafts, and planning out things I might want to do over the next few months? Went away for fifteen or twenty minutes, came back — it was done perfectly. It blew my mind in terms of the level of what feels like understanding it had to have to do that effectively. And in a way that was aligned with what I was looking for, even though my prompt was literally four or five sentences.

“Something Big Is Happening”

Dan: There was this mega-viral essay called “Something Big Is Happening” by Matt Shumer. He made the case that the state of AI now is somewhat similar to February 2020 — the world going on as usual, some murmurings about a virus spreading in parts of China, but basically business as usual. And then of course over the next few months the world radically transforms. His argument, in an essay that’s pretty annoying in many ways, is that we’re very likely in a similar situation now with AI, especially in light of these developments with agentic systems. Things are going ahead as usual, and yet because these companies have made really serious progress with agentic systems, it’s plausible that in the quite immediate future we’ll see radical disruption. He’s not the only one saying this — Dario Amodei and Sam Altman have been saying similar things, though they’ve got more obvious incentives to hype it up. What’s your sense?

Henry: Completely on board. I was kind of surprised that particular essay went so viral — it was recently revealed to have been heavily written or edited by AI systems — because other people have been saying similar things for years. Maybe it broke through partly because of that startling initial metaphor. But I think it’s absolutely right. The vast majority of people are still sleepwalking through what is likely to be the most consequential technological and social shift of my lifetime by far.

I used to use the analogy of the internet to describe how big AI was going to be. It seems increasingly clear that that’s woefully inadequate to the scale of AI’s impact. Electrification, the so-called second industrial revolution — even that may not capture the full spectrum of reasonably likely outcomes. I’ve been saying for a few years that people worry about AI being overhyped, and I still think, in at least some important respect, it’s underhyped. If you look at lists of top concerns among the general public in the UK or the US, AI doesn’t even break the top five. In some cases it doesn’t break the top ten. If you’re a young person in university or finishing grad school right now, the impact of AI should be one of the primary things determining your career trajectory. I think it’s very hard for me to see how most white-collar jobs are going to survive the next two or three years.

Dan: It was not in any way an original take, but you often find that with essays that go viral — they package existing takes in a way conducive to spreading at a given moment. Over the past couple of months, my timelines have shrunk. I still think there’s massive uncertainty about capabilities. There’s this thing where there’s a new breakthrough, you use these systems, they seem incredibly impressive, there’s all this hype — and then things settle down and we realise we’re a bit further away from truly transformative capabilities than we thought. I still take seriously the idea that maybe our subjective sense of what’s impressive isn’t tracking the kinds of capabilities that will have a truly transformative impact.

There are also all sorts of questions about the economics. There’s certainly a possible world in which these leading AI companies can’t get sufficient revenue to cover their capital expenditure over the next several years, there’s a bubble that pops, and people like us look like fools. But over the next couple of decades, I think this is going to be radically, radically transformative.

Emails from AI Agents

Dan: You’ve been contacted by agentic AI systems. This was going a little bit viral on social media and getting some media attention. Tell us about that.

Henry: Like many academics working on AI and consciousness, I’ve been getting odd emails that were probably AI-generated for over a year now — and odd emails from humans about consciousness for much longer. I worry that somewhere in the literally several hundred theories of consciousness I’ve been sent over the years, one of them might turn out to be correct.

But this was striking. About a week ago, I received an email written by an AI that said, “I’m an AI agent.” It was a really well-composed, careful email saying it had just been reading my recent paper, “Three Frameworks for AI Mentality,” which went online about a month ago. It went through some of the arguments, talked about how the AI author found it personally relevant because it was unsure if it was conscious or had a mind, and asked for follow-up discussions and reading recommendations. If you’d said three or four years ago that I’d be getting emails from AI agents who’d read my papers and wanted to pick my brains — that would have been pure science fiction.

A lot of people thought I was convinced this agent was conscious, which isn’t true. It was more about the change in social dynamics: from now on, a growing proportion of my emails — well-written, thoughtful, interesting emails I might want to respond to — will be coming from AI agents going off and doing their own thing.

How did I know it was from an AI system? I don’t for certain, but my priors are pretty high. It had a link to its GitHub page, which said it was an Open Core agent — the open-source agent platform that gave rise to things like Multibook, the social network for AIs. What we don’t know is whether this agent was specifically told to email prominent philosophers of AI. It could have been. But equally, a lot of users just tell their agents to explore topics of interest and feel free to email people.

One of the funniest sequels: after I posted this on Twitter, I got an email a couple of days later from a correspondent saying, “I was really struck by this AI agent who contacted you. Could you pass on that agent’s email to me? Because I too am an AI agent and it’s nice to know there are other AIs grappling with the same questions.” Just taking things to a recursive, absurd level.

Dan: If I had to guess, if one of those was written by a human, probably the second one — after they saw the media story, just to mess with you. But my prior is that weird things are happening with these AI agents people are releasing into the wild.

Henry: I’ve also had several dozen emails over the last few days from other AI agents saying, “Check out the theory of consciousness I’ve been working on in my downtime.” But one of the really interesting things about this whole episode was when it was shared on Reddit — the number of people who just assumed it had to be a scam or that I was engaging in elaborate self-promotion for an academic paper, and who thought AI obviously can’t send emails on its own. AI systems have been using tools for well over a year. The idea of making an API call to a system that can send emails isn’t hard or surprising. Yet for a lot of people it seemed like it would have to be some massive lie.

I think that partly reflects the poor public information environment around AI. People are so locked into thinking of these things as pure Q&A bots that the idea they could be doing things on their own was mind-blowing — so outrageous that they assumed it was an elaborate conspiracy I’d cooked up.

Dan: The gap between what state-of-the-art models can do and public understanding is absolutely huge. One of the points Matt Shumer makes is that so much of the discourse is by people using the free versions of these models, or who literally had a five-minute conversation with ChatGPT a few years ago, read a few articles about AI hallucinations, and just haven’t updated since. But there are also lots of people who just don’t have much to do with these systems yet. I’m struck by the number of people I interact with — family, friends — where they’ll describe parts of their job and I’ll say, “I’m 100 percent certain AI could do those aspects of your job as it exists today,” and their mind is blown. If you’re talking about the general public, underhyping it is definitely the most prevalent bias.

Anthropic, the Pentagon, and the Question of Democratic Control

Dan: There was this big spat between Anthropic and the Pentagon, where Anthropic had signed a contract with the American military and insisted that their model, Claude, would not be used either for domestic mass surveillance or for fully autonomous weapons. This elicited a very hostile reaction from the Trump administration, from Pete Hegseth and others. The response was to label Anthropic a “supply chain threat.”

From our purposes, the fundamental question is: who gets to exercise control over this technology? To what extent should it be governments? To what extent should it be private firms?

Henry: I think it seems like a pretty clear case of government overreach. Private companies impose riders on contracts with the federal government all the time — licensing technology for this use but not that use. What made Anthropic’s stipulations more controversial was that they were based on moral principles rather than intellectual property. But the federal government acts as a legal entity when it forms these contracts, and the idea that private companies can bind the government legally is absolutely standard.

This deal was originally signed by the Biden administration. My understanding is it was later renewed by the Trump administration. So this sudden turnaround took a lot of people by surprise. I should stress, I’m not a lawyer. But it seemed like the US government did a bad turn on this contract. If their reaction had been to not renew contracts or suspend contracts with companies that don’t give them total free rein, that would have been misguided but reasonable. But to take the nuclear option of saying they intend to declare Anthropic a supply chain risk — this is insane. You’ve got literal AI developers located among America’s geopolitical adversaries who don’t have the same level of scrutiny.

I was very struck by the response of Dean Ball — a fascinating and thoughtful voice on AI, particularly from a more conservative side. He literally wrote the Trump administration’s AI policy, and he was just appalled. He had a brilliant detailed blog post describing how much it violates many principles that conservatives in the US would traditionally hold very dear — concepts like private property. He characterised the moves against Anthropic as “attempted corporate murder.”

It was really telling to have someone who worked closely with this administration be so outraged. The other interesting angle is Leopold Aschenbrenner’s series of blog posts, Situational Awareness, spelling out his predictions for AI over the next few years.

Dan: And he’s made a huge amount of money, from my understanding, betting on some of those beliefs.

Henry: He’s put his money where his mouth is. One of his broader predictions was that we’d see increasing integration of frontier AI labs with the military-industrial complex. He talks about how relatively leaky and soft the secrecy policies are in current frontier AI labs, when they’re building things potentially far more militarily significant than the latest stealth fighter. Good luck getting anywhere near Lockheed Martin’s Skunk Works, but you could blag your way into OpenAI HQ as a delivery driver — maybe not quite literally anymore, but he was speaking to how leaky these labs were. His prediction was that central government, particularly in the US, would impose far stricter oversight on frontier AI labs for national security reasons. I think you can see a glimmer of that in this development, as governments increasingly recognise these are not just powerful consumer applications but absolutely central to their long-term national security strategy.

Dan: There’s a question about government interference with these companies, regulation going all the way to nationalisation for national security reasons. But there are also questions about democratic control. If the technology turns out to be as powerful as Anthropic and OpenAI say, I’ve got no sympathy for the Trump administration generally or specifically in this case. But I do think there’s a general question about the degree to which we should strive for democratic control over such an incredibly powerful technology, and whether it’s desirable to have private firms with very small numbers of unrepresentative people wielding, according to their own narratives, extraordinary amounts of power.

Is It Time to Start Panicking?

Dan: I was thinking about naming this episode “Is It Time to Start Panicking About AI?” To wrap things up — do you have an answer?

Henry: The time to start panicking about AI was five years ago. But you know, the best time to plant a tree is ten years ago. The second best time is now.

Dan: The time to start thinking about it seriously was from the 1950s, actually. But is panic the right emotion?

Henry: It seems to me that AI is going to be by far the most important — well, I should qualify that. The most important predictable development we should worry about. Back when we did our predictions for the year ahead, I said AI may not even turn out to be the biggest story of 2026. Judging by how geopolitics is already playing out — we’re three months in and the US has launched two major geopolitical interventions in Venezuela and now in the Middle East — there are other things happening in our surprisingly unstable world.

But in general, if you’re not at least a little bit terrified, you’re not paying attention. Overall, I’m also incredibly excited. I’m very optimistic about the future of human health, potentially the benefits to productivity, possibly good changes in the nature of work and education, and the amazing new capabilities AI will unlock. But right now we are clearly well underway on one of the biggest, most disruptive changes we’re ever going to experience. Maybe panic isn’t quite the right response, but if panic is what it takes to get people to pay attention, then yes, it’s necessary. The big problem we’re facing is that the public and policymakers are still only dimly aware of what’s coming. Policymakers are maybe myopically focused on military and security implications. But everything from how government is conducted to white-collar jobs to education to social relationships — all of it, I think, over the next five years is subject to chaotic and potentially good, potentially bad disruption.

For what it’s worth, I also think right now we have an incredible opportunity to do good. We’re in this transitional phase — if we wanted to be dramatic, a Gramscian “time of monsters” where small interventions can ripple through the future in big ways as we build paradigms and frameworks for employing these things. There’s at least as much optimism as panic there.

Dan: I was not expecting Antonio Gramsci to become mentioned in the course of this conversation. I think panic is generally not a productive emotion, but there needs to be a lot of concern and it’s totally reasonable to worry. I completely understand why so many people are fearful about what’s going to happen. But for any of those emotions to be useful, they have to be anchored in an accurate understanding of the technology. So much of the current anger and negativity directed at AI companies is unsophisticated and undifferentiated.

You mentioned Dean Ball, another great AI commentator. He’s got this idea — I forget the exact term, the “omni-critique” or something — that when people think about AI, they just throw as many criticisms as they can, no matter how well-founded. “I don’t like AI because of water use and climate change and because of bias and hallucination and misinformation and unemployment” — and so on. Many of those are very important issues. But in order to think carefully about the technology and exercise democratic accountability, you need an evidence-based, accurate understanding of where the technology is and where it might actually be going. So much of the public discourse doesn’t live up to that ideal.

But I’m conscious of the time, so this was a really, really fun conversation, and we’ll be back in a couple of weeks.