Most Questions About AI Aren’t About AI

Why the AI revolution is a golden age for polymaths, philosophers, and moderate epistemic despair

Epistemic status: big picture, speculative; the long-form writing equivalent of thinking out loud.

---

Recently, I told a friend I had been reading a philosopher’s writings about AI. He scoffed: “What does a philosopher know about AI?”

I could understand where he was coming from. His reaction illustrates a popular line of reasoning that might be put like this:

• AI is a complex technology.

• To think seriously about AI, one must therefore have deep technical expertise in how this technology works.

• Those who have this expertise are the “AI experts”.

• Those who lack it but write or talk about AI anyway are either amateurs or bullshitters.

This aligns with how many people think about knowledge and expertise in the modern world: Reality decomposes into narrow domains. So, to learn about the world, we should listen to those with certified expertise in those domains.

Most people who think this way allow that other forms of expertise are sometimes necessary. In thinking about AI’s impact on the labour market, for example, we should probably listen to economists. But even here, there is often an assumption that technical AI experts are uniquely well placed to pontificate.

I think that this whole perspective is confused.

Although deep technical expertise in how AI works is indispensable for answering many questions about AI, it is rarely sufficient and often unnecessary. Most questions about AI simply aren’t about AI in a narrow, technical sense. These include not just questions about how AI will transform our societies, but even questions more directly focused on AI itself, a uniquely mysterious technology that invites deep questions about the nature of intelligence, motivation, and agency.

When technical AI experts speculate about these issues, they often draw on extraneous and mistaken beliefs about psychology, sociology, economics, culture, political institutions, philosophy, and much else.

This suggests that AI discourse would improve by engaging a wider range of expertise, but I think that’s only half-right. The AI revolution threatens the existing division of intellectual labour and specialisation that we rely on to make sense of the world. Not only does it throw up novel challenges for which we simply lack mature disciplines, but it may transform so many parts of the world all at once that it undermines a core assumption of modern intellectual inquiry: that reality decomposes into relatively modular domains that experts can investigate independently.

At least, these are the claims that I will argue for here.

I will also extract an important lesson. Framed optimistically: the AI revolution is a golden age for polymaths and philosophers—thinkers who can escape the straitjackets of existing disciplines and engage in creative intellectual synthesis and big-picture speculation. Framed pessimistically: we must come to terms with moderate epistemic despair. As this new technological revolution unfolds, our world and future will become less knowable and navigable.

To Think About AI, You Must Think About Not-AI

Consider, first, the Industrial Revolution.

In one sense, this was a technological revolution that enabled humanity to unleash unprecedented amounts of mechanical power. Nevertheless, as Will MacAskill and Fin Moorhouse point out, most questions it raised weren’t narrow questions about the workings of steam engines or spinning jennies.

The Industrial Revolution transformed everything, including political and legal institutions, the structure of the economy, class and status hierarchies, human psychology, values, culture, gender relations, family structure, personal identity, ideologies, religion, and art, all of which interacted with each other in highly complex ways.

It was a technological revolution, but not only a technological revolution. The great thinkers who grappled with these changes—Ricardo, Mill, Marx, Durkheim, Weber, and so on—didn’t distinguish themselves with fine-grained technical knowledge about the mechanics of steam engines.

The AI revolution will be similar, except that the transformations will likely happen in a much more compressed timeframe.

An Interlude: Transformative AI Is Coming

Many people today are sceptical of this forecast, dismissing it as the hype and bullshit of self-serving tech bros and weird subcultures of “rationalists” and effective altruists. I won’t pause here to explain why this assessment is so misguided (others have already done so at length), except to briefly say this:

The brain is a complex information-processing mechanism. So, it should be possible to build computing machines that don’t just match but exceed human cognitive capabilities. Putting it this way undersells the transformative potential of AI, however, because it pictures AIs that are simply more impressive versions of us. In reality, the space of possible intelligences is vast, and AI systems will also have qualitatively new capabilities, including the ability to be copied and run in parallel, operate at superhuman speeds, and modify their own designs.

These points were already clear to pioneers of computer science and AI in the 1950s, when one could reasonably have been sceptical. But today, we have built AIs that outcompete humans across a wide and expanding range of cognitive tasks, and there are huge economic, military, and scientific incentives to continue this progress. We should therefore expect to build AIs in the coming years far more powerful and capable than those we already have, including not just digital systems but advanced robots. And eventually, we will build machines more capable across all domains than the most capable humans.

These points hold even if one thinks that the abilities of advanced AI systems today are highly “jagged”, that the current AI paradigm will hit a wall, that we are in a financial bubble, that concepts like “AGI” (artificial general intelligence) make little sense, or that AI’s impacts will be slowed by human, organisational, and institutional bottlenecks and speed limits.

In other words, although one can reasonably debate how close we are to transformative AI, what forms it will take, and how quickly it will diffuse through our economies and societies, we should be highly confident that it is coming, and take very seriously the possibility that it will arrive within our lifetimes.

Even Many Questions Directly About AI Aren’t About AI

The Industrial Revolution also highlights another, subtler point: the core technologies themselves often outran humanity’s scientific understanding of why and how they worked. The attempt to understand steam engines, for example, played a major role in the development of thermodynamics, which emerged long after steam engines had begun transforming the world.

A similar but more extreme situation arises with AI.

On one level, this is just the familiar point that even technical AI researchers lack a precise, mechanistic understanding of why and how deep neural networks work. However, this observation greatly understates how much more mysterious AI is than any technology that has come before. Even if we had satisfying mechanistic explanations of how ChatGPT or Claude work, how we should think about AI would still depend on a vast range of questions about the nature of minds, intelligence, and agency that would remain largely unsettled by this knowledge. Moreover, these questions are highly relevant for understanding how the AI revolution is likely to unfold.

Consider just some of the controversies that dominate discussions in this area:

• Will “super-intelligent” AI systems kill or disempower humanity?

• How easy is it to “align” AI systems with human values?

• What capabilities must AIs have to match human cognitive performance across all domains?

• What can superhuman reasoning power enable a system to achieve in the real world?

• Does it make sense to talk of intelligence as a single scalar property that systems have more or less of?

• Is the concept of “Artificial General Intelligence” useful or even meaningful?

• How much room above the peak of human performance exists for different capabilities, such as persuasion and forecasting?

• How should we measure the rate of AI “progress”?

• Could current or future AI systems have conscious experiences?

These are questions directly about AI, much more so than questions about how AI will transform our relationships, economies, and cultures. But although technical AI expertise is relevant to many of them, they go far beyond technical matters.

Thinking About (Thinking About) AI Doom

Consider just the topic of AI takeover, the popular threat model according to which advanced AI systems will eliminate or disempower humanity. Although some discussion of this risk is informed by technical details about how modern AI systems work, much of it is driven by high-level theoretical assumptions, analogies, and thought experiments.

For example, the most influential argument in this area draws on the concept of “instrumental convergence”, an alleged tendency of intelligent systems to converge on instrumental goals like self-preservation and power-seeking for a wide range of possible ultimate objectives. This is taken to demonstrate that systems tasked with seemingly benign objectives like creating paperclips might end up pursuing instrumental goals that conflict with our interests and even survival (e.g., using our bodies as raw materials for paperclip factories).

Two other influential arguments allege, first, that “aligned” or “friendly” super-intelligent systems comprise a tiny fraction of possible super-intelligent systems, implying that misaligned AIs are much easier to build than aligned ones, and second, that there are in-principle difficulties in training machines to be aligned instead of merely appearing aligned until they accrue more power.

These arguments attempt to derive conclusions from first principles about the nature of agency, intelligence, and learning, and they were advanced before modern large language models were invented.

Of course, one might reasonably respond that things have changed today: we now have many technical AI safety researchers and an extensive empirical literature that bears on these worries, including clear demonstrations of “alignment faking”, deception, and other worrying behaviours in AI models.

Nevertheless, the highly theoretical arguments advanced by figures such as Eliezer Yudkowsky and Nick Bostrom continue to play a huge role in shaping how even AI safety researchers interpret specific findings and frame threats. For example, whether researchers treat an AI hacking into a private company as a rare, fixable outcome of current training methods or as a harbinger of future doom depends largely on their pre-existing beliefs. That is, the interpretation of the findings is as much informed by pre-existing theory as the theory is informed by the findings. This explains why researchers exposed to the same findings can disagree so much about the likelihood of future AI takeover, in contrast to fields such as climate science, where high levels of expert consensus reflect the much greater role of rich bodies of empirical data and decades of confirmed empirical predictions.

If you’re a hard-nosed empiricist who thinks knowledge should be rooted in empirical findings and peer-reviewed scientific consensus, and who scoffs at elaborate first-principles reasoning and speculation about complex systems, you will probably not be very impressed by these arguments.

In contrast, the self-styled “rationalists” most worried about AI doom, such as Yudkowsky, have an extremely optimistic view of the powers of pure reasoning. Hence, they are happy to reach high levels of confidence in bold forecasts even when direct data is sparse, based on complex chains of abstract theoretical reasoning alone.

Whatever one thinks about these disagreements, they further illustrate my point. One of the central questions about AI connects not only to high-level theoretical issues about intelligence and agency but to epistemological debates about the powers of pure reasoning.

In fact, one of the most fascinating aspects of the AI takeover debate is how these epistemological disagreements feed back into how people think about the power of AI systems themselves. The rationalists most worried about AI takeover are so worried, at least in part, because they assume that a machine with superhuman intelligence and reasoning abilities would have immense power, including the ability to take over the world and wipe out all of humanity.

In contrast, those who are sceptical of the power of speculative rational argumentation to deliver substantial knowledge about the world are also typically sceptical that merely endowing machines with more cleverness and reasoning ability will automatically endow them with real-world power.

In other words, our beliefs about what we humans can accomplish through pure reasoning feed into our expectations about how powerful pure reasoning will be in machines.

On Coping With Technological Revolutions

An obvious corollary of all this is that technical expertise does not, itself, confer expertise, or even special insight, on many of the huge challenges raised by AI. We can’t simply defer to “AI experts”.

The classic illustration of this point is Geoffrey Hinton’s 2016 recommendation to stop training radiologists on the grounds that AI systems would soon outperform them at image classification. Hinton, a “godfather” of AI, was highly prescient about trends in AI capabilities. But he coupled this expertise with a low-resolution folk model of work and labour economics. Economists will tell you that, like almost all professions, radiology involves a complex, integrated bundle of tasks and responsibilities, and that radiologists operate in a real world in which technological diffusion is slowed by regulations, liability regimes, and numerous other frictions and speed limits.

Ten years later, not only do radiologists still exist as a profession, but demand for their work is apparently rising.

Many philosophers and neuroscientists would make similar complaints about Hinton’s arguments concerning AI consciousness, and many experts on persuasion and belief formation cringe at some of Hinton’s forecasts about AIs’ superhuman persuasion abilities.

One simple lesson you might draw from these reflections is that to grapple with the AI revolution, we need to engage with a wide range of academic disciplines. Computer scientists won’t suffice; we also need economists, psychologists, sociologists, political scientists, philosophers, and so on.

In some sense, this is obviously right.

However, this response can also fail to grapple with the severity of the epistemic challenges ahead. Appeals to the importance of multiple forms of expertise often assume that problems can be divided into existing disciplinary buckets, that relevant knowledge already exists within these disciplines, and that we can address these problems by simply integrating such knowledge.

These assumptions are likely too optimistic.

A New World Requires New Knowledge

Consider questions that seem absurdly science-fictional to many now, but will soon acquire monumental importance: How should we think about institution design, ethics, and social justice in societies that contain vast numbers of digital minds and robotic agents? How should we understand the collective dynamics of agentic AI “swarms”? What happens to science, research, and technological development when AI systems can substitute for much of human cognitive labour?

For these and countless other questions, many of which we haven’t yet even anticipated, bodies of knowledge, concepts, and methods across existing fields of research will obviously be important. But it should be equally obvious that these questions throw up fundamentally novel kinds of puzzles and challenges that existing disciplines aren’t well-equipped to handle. The knowledge we have accumulated about individual and collective behaviour has targeted human agents (or, at most, organisms). Even when we have constructed more idealised models of rational agency, coordination, competition, conflict, and institutions, they have been rooted in our experiences of human agents.

Advanced AIs won’t just have a wide range of unprecedented capabilities in things like copying, parallelism, processing speed, high-bandwidth communication, and self-modification. They will also have new kinds of goals and motivations (if those terms are even applicable) shaped not by ruthless Darwinian evolution but by complex mixtures of intentional design, curated training data, human feedback, and economic and geopolitical pressures.

At present, we simply lack mature sciences of what might happen when millions or more artificial intelligences with superhuman capabilities and deeply inhuman goals enter our societies, cultures, economies, and institutions. To make progress on the many explanatory and normative questions this will raise, we will need new concepts, models, theories, and fields.

The Prospects of Decomposition and Specialisation

A deeper issue concerns the prospects for intellectual specialisation under conditions of radical technological transformation.

As with wealth creation, the advancement of human knowledge is closely connected to a complex division of labour. If we had to create all our knowledge individually, we would know almost nothing. So, we specialise in acquiring narrow forms of expertise and then share the fruits of our intellectual labour with others, enabling cumulative advances in the frontier of knowledge and understanding over time.

Modern science and academic research formalise this division of labour. To simplify greatly: economists study the economy, political scientists study political institutions, sociologists study social relations, and so on.

Of course, people often complain about this modern balkanisation and siloing of research. Don’t most big questions straddle multiple disciplines? Why can’t we go back to the good old days before such extreme intellectual specialisation, when thinkers like Smith or Weber or Durkheim engaged with huge questions about humanity, society, and culture?

In some ways, these concerns are legitimate, but they are also naïve. If we want to understand the world, there is simply no alternative to narrow specialisation and the division of intellectual labour. Figures like Smith or Marx could pontificate about huge, general questions because they were writing when nobody really knew anything. Today, we have built mature research fields across many narrow domains, and to the extent that interdisciplinary research is even possible, it is by integrating the fruits of this specialisation, not by replacing it.

An important question, though, is why this specialisation works. After all, society really does seem to be a vast, unimaginably complex, integrated whole. How can people make progress in studying any one part without also engaging with the broader system?

In a classic article on the “architecture of complexity”, Herbert Simon provided an important answer: complex systems become comprehensible when they involve a kind of near-decomposability—subsystems with strong internal interactions but weak interactions with each other, at least over short timescales. These weak interactions mean that other aspects of the broader system can be, to some extent, bracketed off as stable background conditions. That is, although highly specialised inquiry doesn’t require independent domains, it does require interactions between those domains to be sufficiently weak or slow that researchers can treat the rest of the system as relatively constant or “exogenous”.

The AI revolution may threaten this condition. As advanced AI systems and robotic agents are rolled out, they may transform many parts of the social system all at once, eroding the decomposability on which disciplinary specialisation depends. Economic changes may transform political power and public opinion; these changes may reshape ownership and regulation; changes in the information environment may affect democratic preferences; and all of these changes might feed back on one another and countless other forces, including the trajectory of AI itself. Such disruptions needn’t be a permanent condition of our future, but deep technological revolutions can create a transitional period in which so many aspects of society change so profoundly, and interact so strongly, that background conditions specialists hold fixed no longer remain stable.

An Illustration: The Economics of Transformative AI

Consider, for example, one of the most important questions about transformative AI: how will this technology affect the economy? Should we expect mass unemployment? Will it unleash unprecedented amounts of economic growth? How will the concentration of wealth be affected?

Famously, economists and AI specialists are often exasperated with each other on these questions. Economists complain that technologists routinely ignore elementary concepts within economics, such as comparative advantage, bottlenecks, and cost disease, whilst technologists respond that economists treat AI as just another technology, ignoring the very features that make it so transformative, including its potential to one day substitute for all human labour.

One response to this kind of disagreement is to plug truly transformative AI into standard economic models and explore what happens. A fascinating and important literature now does just this, treating AI as a kind of reproducible capital that can replace human workers across all tasks. As you might expect, the models often output insane conclusions: once growth rates are no longer bottlenecked by the population of human workers, they can explode, alongside huge increases in inequality and the near-disappearance of labour’s share of income.

We absolutely need more of this kind of work, and the insanity of its outputs is a point in its favour. We should expect a crazy future.

And yet, the models picture worlds in which transformative AI can simply be inserted whilst a whole range of background conditions—property rights, ownership structures, firms, and so on—remain fixed.

This seems very implausible. One needn’t be a Marxist to appreciate that profound technological change (changes in the “forces of production”) is very unlikely to leave the core political-social-ideological structure intact.

Transformative AI won’t just be an input into our existing system of bureaucratic capitalism any more than industrial technologies were merely an input into the economic structure of the agrarian and feudal economies that preceded it.

In periods of relative stability, we can treat “the economy” as a stand-alone object of study, bracketing off the political institutions, laws, values, incentives, and public opinion that underpin its existence. In deep and wide-ranging technological revolutions, this stability breaks down. In this sense, just as most questions about AI are not only about AI, so most questions about AI and economics are not only about AI and economics.

A Golden Age For…

There are many lessons one might take from these reflections. I will briefly mention just three, although they are really three framings of the same lesson.

… Polymaths

First, the AI revolution will be a golden age for polymaths. If the most important questions it raises cut across existing disciplines and often reach far beyond them, it should increase the value of theorising and research that escapes the constraints of narrow academic specialisation, synthesising ideas and findings from different domains, and proposing fundamentally new concepts, frameworks, and even academic fields.

Admittedly, this kind of activity carries obvious risks. In some sense, true polymathy is simply impossible today. Researchers can’t even fully master the body of knowledge in their own fields, let alone multiple ones. And in practice, lots of “bold” intellectual syntheses and speculations are amateurish and pointless.

Still, if the world transforms in ways that don’t respect existing academic specialisations, we must confront the epistemic challenges this creates.

… Philosophers

It will also be a golden age for philosophers.

In some sense, it’s obvious that philosophy is relevant to many of the questions raised by AI, from questions about machine consciousness and moral status to debates about what a just society looks like in a world with transformative AI. Historically, philosophy and AI have always been closely entangled. Alan Turing’s classic 1950 “Computing Machinery and Intelligence” was published in Mind, one of philosophy’s flagship journals. It begins with the question, “Can machines think?”

Philosophy’s relevance to the AI revolution is even broader than this, however. As a discipline, it is fundamentally concerned with important questions that aren’t yet owned by mature scientific disciplines, typically because they touch on deep conceptual and theoretical puzzles that we haven’t solved or involve ethical or political issues that science alone cannot settle.

Consider two questions: “How does the visual cortex in the brain work?” and “Are shrimp conscious?” The first is a question for neuroscience; the second is more recognisably philosophical. This isn’t because neuroscience and other forms of scientific research aren’t relevant to the latter question. They’re highly relevant. But we don’t—yet—know how to turn the question into a tractable one that might be answered by ordinary scientific methods. It is connected to difficult conceptual, theoretical, and metaphysical questions that humans remain deeply confused about.

People sometimes ask why, unlike science, philosophy doesn’t make progress. This brief reflection illustrates why the question often rests on a semantic mistake. Once a discipline starts to make straightforward scientific progress, we simply stop calling it “philosophy”. Philosophy is our oldest discipline. Pretty much all questions about reality and our place in it were once philosophical questions. Over time, we have turned many of these questions into tractable scientific puzzles and research programmes, giving rise to fields like physics, biology, and psychology. Philosophy is largely the questions that remain.

AI will raise the value of philosophy in this sense. At present, no observations, experiments, statistical analyses, or scientific models could straightforwardly answer how we should live alongside artificial minds, whether advanced AI will disempower humanity, or what the state and economy will or should look like when we have AIs that can outcompete humans at everything.

This doesn’t necessarily mean a golden age for professional philosophy, which can be as myopic as any other area of modern research. To be useful, philosophy must be non-dogmatic, informed by science, and engaged with real-world challenges. Much professional philosophy today doesn’t meet those standards, and much of the best philosophy that does isn’t done by professional philosophers.

Philosophy is an activity, not a guild.

… And Epistemic Despair

A golden age for polymaths and philosophers is not a golden age for knowledge and understanding. If anything, the reverse is true: bold intellectual syntheses, creativity, and philosophising rise in value precisely as the world becomes more chaotic and confusing.

A division of intellectual labour into specialised fields is a major reason modern science is so effective. It makes learning about the world a relatively simple matter of trusting what specialists in specific domains agree on.

Admittedly, this epistemic order performs worse than many would like to acknowledge. The feeling of understanding often outruns its reality. We are much better post-hoc storytellers than forecasters. But still, if we step back and reflect on how much we can know today, it is a remarkable success story.

The AI revolution threatens it. Not only are the certified experts not experts on most of the questions that matter, but it’s often unclear who even would be. We lack mature sciences for many of the unprecedented challenges and opportunities we will confront. The technology will affect so many different aspects of society at the same time that society’s current decomposition into modular parts may break down, at least before a new decomposition—a new regime—emerges from the ashes.

Narrow knowledge will still be attainable: how specific AI systems perform on benchmarks, how workers use them, whether they are more persuasive than human debaters, and so on. But as we move from these local, technical issues to more general, high-level questions about the coming transformations, our confidence should decrease sharply.

***

Writing about AI’s impact on the economy, Derek Thompson says, “I can’t emphasize enough that ‘nobody knows anything’ is about as close to the reality here as three words are going to get you”:

Nobody [knows] what’s going to happen this year, or next year, or the year after that. There is no secret cigar-filled room of people who have unique access to some authentic postcard from the future. When you drill down underneath the bluster, the boosterism, the fear, the anxiety, what’s there at the bottom is genuine uncertainty, a vacuum into which storytelling is flooding. The frontier labs don’t really know what they’re building exactly, and economists don’t really know how to model the thing they claim they’re building (genuine recursively self-improving AI agency isn’t really analogous to something we know about).

“Nobody knows anything” isn’t quite true. Beyond the narrow questions already mentioned, I think we can know at least one very big thing with a high degree of confidence: transformative AI is coming, and maybe soon, even if its shape, timelines, and trajectory are mired in radical uncertainty. Moreover, not all speculation and opining are equally valuable. We are all ignorant, but some are more ignorant than others.

Still, Thompson is clearly onto something. The disagreement, confusion, and storytelling that characterise AI debates today aren’t just a product of ordinary human biases and shoddy reasoning. The future itself is largely opaque. And if this feels true now, as we sit in the foothills of the singularity, try to picture the epistemic vertigo we will experience once the AI revolution really gets going.

Return to my friend’s question: “What does a philosopher know about AI?”

Not a lot, and certainly not enough. In that sense, philosophers are in good company.