What is the likelihood that we will reach artificial general intelligence (AGI) by 2030? What if the rapid progress we have seen in recent years doesn’t just continue but accelerates, or even explodes? How should we think rationally about such possibilities?
In this episode, Henry and I speak with Benjamin Todd, co-founder of 80,000 Hours and author of the new book 80,000 Hours: How to Have a Fulfilling Career That Does Good. Benjamin has become one of the best writers on AI, producing highly informative essays such as “Will we have AGI by 2030?”, “How AI-driven feedback loops could make things very crazy, very fast”, and “How not to lose your job to AI”.
Among other things, we discuss:
Why Benjamin thinks artificial general intelligence by around 2030 is a serious possibility.
Why the most important question is not just whether models get bigger and better in general, but when new feedback loops kick in.
The difference between impressive benchmark performance and real-world economic usefulness.
Whether artificial intelligence research is easier to automate than ordinary white-collar work.
Chip bottlenecks, data centres, talent constraints, and the geopolitics of Taiwan.
Whether artificial intelligence will cause mass unemployment, and why “become a plumber” is probably not good career advice.
Why 80,000 Hours has shifted so much of its attention towards artificial intelligence.
Alignment, control, concentration of power, engineered pandemics, and other risks.
Whether effective altruism and 80,000 Hours are making a dangerous reputational bet on short artificial intelligence timelines.
The worry that much short-timelines thinking comes from a small, socially connected, and homogeneous intellectual community.
This was one of my favourite conversations on this podcast so far. It’s pretty dense and high-level in parts. (I would recommend reading some of the essays we link from Benjamin if you struggle to follow parts of the discussion). But if you’re interested in hearing some of the most persuasive, evidence-based arguments for why the world might be about to become extremely crazy, extremely fast, I highly recommend it.
Links and further reading
80,000 Hours: How to Have a Fulfilling Career That Does Good — Benjamin’s new book.
“Will we have AGI by 2030?” — Benjamin’s case for taking short timelines seriously.
“How AI-driven feedback loops could make things very crazy, very fast” — the most directly relevant essay for the “explosive progress” theme.
“How not to lose your job to AI” — Benjamin on careers, automation, and which skills may become more valuable.
“Shortening AGI timelines: a review of expert forecasts” — useful background on how forecasts have changed.
“Are the last 3 months the start of an AI acceleration?” — Benjamin’s more recent reflections on whether progress is already speeding up.
Dwarkesh Patel, “Thoughts on AI progress” — includes the line we discuss in the episode: are models getting more impressive at the rate short-timelines people predict, but more useful at the rate long-timelines people predict?
Matt Reardon, “If Anyone Bursts It, Effective Altruism Dies” — a sharp piece on artificial intelligence hype, effective altruism, and reputational risk.
AI 2027 — the influential scenario forecast that has shaped a lot of recent discussion about short timelines and intelligence explosions.
80,000 Hours, “How to get into AI safety in 3 months” — a practical guide for people who want to work on reducing risks from advanced artificial intelligence.
Transcript
Please note that this transcript has been lightly AI-edited and may contain minor errors.
Introduction
Dan: Welcome back. I’m Dan Williams, back with my co-host Henry Shevlin. Today we’re honored to be joined by Benjamin Todd. Benjamin is co-founder of 80,000 Hours, an enormously influential organization dedicated to helping people have the most impactful but also rewarding careers possible over the roughly 80,000 hours that make up a typical working life. He’s also the author of a brand new book conveniently titled 80,000 Hours. If you’re watching this, you can see me putting it in front of the camera. I highly recommend that people buy it and read it. And finally, he has one of the very best Substacks in the world for learning about AI, the key developments and risks, and how to navigate our transition, both as individuals and as societies, to a world after truly, radically transformative AI, which is going to be the primary focus of our conversation today. Benjamin, welcome to the podcast.
Benjamin Todd: Thanks so much for having me.
Dan: The thing we really want to focus on is how you’re thinking about AI, how things are going to develop over the next five or ten years, and how we should navigate that. But it would be helpful to start with a brief primer on what 80,000 Hours is, why you started the organization, and why you’ve recently published this book.
What 80,000 Hours does, and the pivot toward AI
Benjamin Todd: The idea is that your career is the biggest decision you’ll ever make, especially for your impact on the world. It’s your biggest resource for helping others and doing good. But there’s very little advice on which careers are actually worth taking in the first place. Most careers advice is just about how to apply to consulting jobs, or how to become a lawyer, but not really which paths are worth going down in the first place. So there’s this enormous opportunity that maybe millions of people are squandering. 80,000 Hours tries to solve that by providing free careers advice. We have online research, a podcast, a job board with a thousand open opportunities, and one-on-one advising.
Dan: I think of it as a great example of evolutionary mismatch. Up until about five minutes ago in human history, the idea that you’d need to think rationally about what to do with your career wouldn’t have made much sense. Now we’re in conditions where it’s hugely important, but we can’t just rely on evolved adaptations to make those decisions rationally, so you need this cultural technology of 80,000 Hours to guide people through it. Another thing worth flagging before we segue into AI: my sense as an outsider is that recently 80,000 Hours has really pivoted to focus centrally on AI, away from the more generalist advice package it was giving before. First, is that impression correct? And second, what drove that change?
Benjamin Todd: It’s partly correct. In the book, a lot of the advice is totally general-purpose careers advice. It’s about the heuristics and frameworks to use when choosing between options, and very practical things like how to actually do a job application and interviews. So that applies to everyone. But then there’s a chapter where we talk about which problems are most pressing in the world, where we present our own view. And for things like the one-on-one advice, because that’s very expensive to deliver per person, we tend to focus it on the problems we think are most impactful to help people enter. So it depends on which part of the organization. But in general, yes. We’ve had AI risks at the top of our list of most pressing problems since before 2016, so for more than ten years.
One big reason is that the timeline to AI has got shorter and shorter, so it’s become a greater and greater priority as that’s happened. The other big reason is that, although it’s a lot less neglected than before, it’s still extremely neglected compared to most other problems people talk about. The number of people working on AI alignment risks is maybe one to three thousand people in the world, even though that might be one of the biggest existential risks we’re ever going to face. And then there are all these other AI risks that have come up recently, like concentration of power, what we’re going to do about digital sentience that you had a great interview about, gradual disempowerment, and so on. These literally have like five or ten people working on each one. So you might think that as AI gets closer it becomes less neglected, and it kind of does, but not enough to offset the huge increase in urgency.
Henry Shevlin: It might come as a surprise to some listeners that most people in the world are not really concerned about AI, given how unrepresentative our listener base probably is. But I completely agree, Benjamin: if you look at the top five concerns in pretty much every Western democracy, AI almost never makes the cut. Very few people are working on this, even though the three of us and our listeners live in these bubbles where 80% of our news content is about AI. Before we drill down on AI, I wanted to ask one other quick thing.
Personal fit, rare skills, and choosing a career
Henry Shevlin: For a lot of people, intuitively, when it comes to choosing a career there’s a tendency to prioritize rare skills. If I could be the best medieval-style flute player in the world, and I’ve got a rare talent for that, there’s a sense a lot of people feel that it would be a terrible shame not to use it, even though the actual positive impact, financial earnings, and maybe even happiness associated with it are less than something more mundane, like becoming a solicitor or an accountant. I’m curious whether you think that’s a correct diagnosis of one of the problems people face when choosing careers, and whether you think there’s any reason to prioritize rare skills.
Benjamin Todd: We talk about personal fit, and there’s a chapter about this in the book. You can think of it as just how good you’ll be compared to the average in a certain career path. In a way it’s the most common advice, the advice my dad gave me, which is “do what you’re good at.” But despite that, people do still somewhat underrate the importance of it. Partly there are these heavy-tailed differences in outcomes. There’s a famous study of expert performance, I think by Simonton, and he finds that something like 50% of the contributions in a field come from the top 10% of contributors. So there are huge differences in people’s fit between different career paths.
The second point is that if you’re at the top of a really impactful field, obviously that can be really impactful. But even if your field isn’t directly impactful, just being successful at anything tends to give you a lot of options to have an impact indirectly. That could be through donating, or changing the conversation about an issue, or meeting other influential people and spreading ideas through them. These indirect ways of contributing tend to get totally neglected in the normal ethical careers advice, which is basically that you should do a helping career, like being a doctor, a teacher, or a charity worker, and that those are the most impactful jobs. But to a large extent it’s much more about how you use the position you have than the specific job title. That said, there is a trade-off. Some careers do give you more opportunities to help than others. If you’re a super niche medieval historian, there is definitely a trade-off there. So you basically have to weigh the two, and it depends on what other options you have.
The case for AGI by 2030: four drivers of progress
Dan: If people are interested in the general 80,000 Hours package, how to think about this in a rational, evidence-based way, there are loads of resources online, plus the book. Let’s pivot to AI. We can come back later to how thinking about what’s going to happen should influence people’s career decisions. You mentioned that timelines to transformative AI now seem alarmingly compressed. It all seems quite imminent in a way it maybe didn’t ten or fifteen years ago. You’ve got a very interesting essay, called different things online, but on your Substack I think it’s “The Case for AGI by 2030.” Walk us through the argument.
Benjamin Todd: There are a lot of different ways of thinking about timelines. The one I take in this article is essentially trend extrapolation, but taking an extra step and asking what the underlying drivers of these trends are and how long we should expect them to keep working. I break it down into four main drivers. There’s increasing pre-training. There’s doing more RL, reinforcement learning. There’s doing more test-time compute, so running the models for longer. And then there’s building these agent scaffolds and having the models go and complete real-world tasks, where you’re also doing RL on whether they succeeded or not. Then there’s an underlying thing driving those four, and potentially new drivers, which is the scale-up of computing power and research labor going into AI.
The basic argument is that all four of these drivers are set to continue, as are the underlying drivers of compute and labor. So we should expect AI progress to continue pretty rapidly, maybe at a similar pace to recently. There are some reasons to think it could accelerate further, and some reasons to think it will eventually slow down, though I’d argue that’s probably after 2028 before it kicks in. We still don’t know where it will end up or how good it will get in a given time frame. But you can at least build a case that there will be large further advances, and it’s plausible you get to something like pretty close to AGI itself, or close to an AI that can help accelerate AI research, or just an AI that can produce enough revenue to keep the whole machine going so that you eventually get there.
I don’t necessarily think it will be before 2030. I’ve actually changed the title now to “Will We Have AGI by 2030?” I do think that’s a very realistic possibility, though I also think it’s very realistic that it’s at some point in the 2030s. And there’s still some chance it’s even beyond 2040 or beyond 2050, though that seems relatively less likely to me at this point.
Henry Shevlin: Out of interest, how do you operationalize the arrival of AGI? A lot of seemingly clear things, like the Turing test, are kind of gone now, but it’s probably not going to be one moment we can look back on and say “that was the day the Turing test fell.” Do you think AGI is going to be similar, or is there a clear landmark?
Benjamin Todd: No, I think it is just a spectrum. I have a post about this on Substack. This came out of DeepMind and Shane Legg. One way of defining AGI is that there are all the possible tasks you can do, and then there’s how well you can do each task, which is the strength of your capabilities, and then there’s how broad a range of tasks you can do, which is the generality of your capabilities. AGI is just AI that’s becoming more general and more capable. Where you eventually draw the line and say “this is AGI” is arbitrary, though people normally draw it around the level of a human worker: you can do about the same range of tasks about as well as a human worker, for the same cost.
The reason people choose that is because once you automate human labor you can get a type of feedback loop that could lead to transformative change of the economy. Though there are other points at which you might get transformative change well before that. So from a day-to-day planning point of view, I actually focus more on an AI that could automate AI R&D itself, which I’d guess would come significantly before that, maybe only one or two years before, but earlier in the process than an AI that can do almost all jobs. That’s because doing almost all jobs requires dealing with loads of legal issues, social jobs, and physical jobs, which all look further away than purely virtual remote jobs.
Pre-training versus the newer drivers
Dan: I want to come back to AI systems that can substitute for human labor in the specific domain of AI R&D. But on those four drivers, one thing I’m confused about is the first one, pre-training. As I understand it, building bigger models and throwing more compute at them during pre-training is where a lot of the well-formulated scaling laws applied, and where we got rapid advances in capabilities. Then I came across a narrative that there are now pretty sharp diminishing returns and we’re not getting the improvements people hoped for from pre-training specifically, so we increasingly rely on these other domains like reinforcement learning, agent scaffolding, and test-time compute. What’s your sense of that? Are we still seeing really big returns from scaling pre-training, or are we now more reliant on the other stages?
Benjamin Todd: There’s definitely been a huge shift toward the other drivers. Before, pre-training was basically the only driver. In some of the most recent models, though we don’t really know, it’s like half spent on reinforcement learning and half on pre-training. I see that as the labs realizing they could get much higher returns from the other drivers, partly because they were starting from a really low base, so it made sense to invest in those to catch them up. But the people at the companies all tell us that more pre-training is still helpful. This is one of the best guesses for why Mythos was above trend on a bunch of benchmarks: they ramped up pre-training a lot. It was the first big new model we’d had for a while, and they got one or two years of pre-training progress in one leap, so it went above what looked like the trend line.
Dan: That’s interesting. Another question about the trend extrapolation you’re doing here: you’re talking about ways these systems are being made better and the factors that enable that, through revenue funneled back into training runs and increasing research labor. But there’s also a question about how we measure what it means to say they’re getting better, and how we track that. They’re certainly getting way more impressive, and in certain domains, like coding, much, much better even since you wrote that post. And there are various benchmarks across domains where they seem to be getting better and better.
An argument you often hear from the other side is: okay, you’re making progress in domains like maths and coding where you have verifiably correct outputs, and they are getting subjectively more impressive, but if you focus on more objective measures, like what percentage of human economic activity is actually being performed by automated AI systems, or the extent to which markets are pricing in really transformative AI, you get a different picture. There’s a classic quote from Dwarkesh from last year, something like: models are becoming more impressive at the rate the short-timelines people predict and more useful at the rate the long-timelines people predict. How are you thinking about that?
Benchmarks versus real-world value
Benjamin Todd: There’s a lot of good stuff to dig into here. On the last thing you said, that felt very true to me last summer, and a lot of people lengthened their timelines then. But as of Q1, with the Claude Cowork boom, that stuff really started working in a way that seemed like a new regime.
I think this is one of the most crucial issues in timelines: how to translate benchmark progress into real-world tasks. That’s probably the best case for AI skepticism. It’s something like: AI will become really good at very measurable things like coding and maths, we’ve got a couple more years of that, and then it becomes too hard to keep scaling compute, because you can’t make unlimited revenue just with these very verifiable domains. So the whole thing slows down and we end up with these incredible but still somewhat narrow tools, a long way from a general-purpose agent that can actually run a company, which is what you’d need to go into some new, faster rate of economic growth or scientific progress.
That said, I do think the evidence is that they’re improving very rapidly at these other tasks. My favorite benchmark, in a way, is just revenue, and revenue is growing super fast. Anthropic and OpenAI are up to around a hundred billion of revenue now, and for the last four years it’s been growing more than three times per year. People say it’s an S-curve and it’s going to plateau, and maybe, but actually it’s been accelerating further. So far this year it’s been growing at eightfold per year, so it actually looks like a super-exponential trend where we’re still on the up part. And you don’t need to project many more years into the future before you’re getting to tens of trillions of revenue, which is roughly the amount you’d expect from automation of large amounts of remote work. Ultimately, “will someone pay for it” is a very hard metric to game, and that picture also suggests very rapid progress and maybe quite short timelines.
Where are the productivity gains?
Henry Shevlin: Do you think we’re starting to see this in the productivity figures yet? There’s this famous observation by Robert Solow about the computer age, that you can see it everywhere except in the productivity figures. I’ve seen people raise similar worries about AI: we’re spending vast amounts of tokens, but where are the incredible new software programs that we didn’t have two years ago that should now be flourishing and proliferating?
Benjamin Todd: This is quite a confusing thing. A hundred billion of revenue is very big in one sense, but it’s also still very small compared to the world economy as a whole. Global GDP is about 130 trillion, so you’re still at only 0.1% of world GDP, which you’d actually expect to not be that noticeable. Where this really becomes a big deal is a few more years in the future, when you’re getting up to three or ten percent of GDP. That’s when we’d really be noticing it in our daily lives.
Why progress might level off after 2030
Dan: Let’s double-click on a couple of things. The central thesis of the article is about whether we’re going to get to AGI by 2030 or thereabouts. There’s an argument that if we don’t get there by then, it’s probably going to be much longer than you might intuitively think, because we won’t be able to keep scaling the amount of compute. The intuitive view would be that if we don’t get it by 2030, maybe we get it by 2032, and so on. But you’ve got a somewhat different model, precisely because it’s so dependent on scaling these inputs. Would you mind walking through that aspect of the argument again, and whether your views on it have changed since this came out last year?
Benjamin Todd: Totally. One quick thing on the last point first: I do think a lot of AI value will come up as consumer surplus, and a lot of that will be quite hard to measure as GDP. So I think we’ll end up in a situation where there’s actually a lot faster growth, in some sense, than it looks in the official statistics. That could lead to interesting situations where people think inflation is way higher than it really is, and then you have policy based on incorrect stats. But that’s a whole other topic.
On the leveling-off thing: recently the amount of computing power, the number of chips produced each year, has been roughly doubling every year. They also become 30% or 40% more efficient. So the total amount of compute is tripling every year, and the amount controlled by OpenAI and Anthropic is growing even faster because they’re also becoming a larger share. That’s driving very rapid progress, because those are insanely fast trends, way faster than Moore’s Law or pretty much any other trend we deal with. But this has been possible because we have all this chip production capacity that has been transferred over to making AI chips. Most of the advanced AI chips are produced by TSMC, this one company in Taiwan. Even now, the most advanced nodes, the name for the most advanced production capacity, still mostly go to smartphones, to your Apple smartphone, because that’s where you need the most efficient, smallest, lowest-energy chip possible. But now we’re getting to the point where it’s almost around half AI.
If you have one more doubling, then pretty much all of the most advanced capacity is being used on AI. At that point they have to start producing new chip fabs, which is the name for the factories that make chips, and that’s significantly slower to do than just converting existing capacity to a different type of chip. One idea is that we eventually get bottlenecked by the production of lithography machines, maybe the most complicated machine in the world, which you need to make a new fab. Dylan Patel of SemiAnalysis thinks that will be the hardest thing to produce more of in the whole production chain, so the production of those eventually becomes the bottleneck. That could lead to a situation, and I still need to make the spreadsheet of this, where production is something like 30%, 40%, or 50% growth per year rather than doubling or more. We don’t know how fast it could speed up once all of civilization is focused on building this one machine, because it’s the key bottleneck to all AI progress, but at least some people think you might not actually be able to produce that much more of it.
So that’s the main reason. And one thing in my piece is that it probably doesn’t actually stop. As long as you’re still getting enough revenue to pay for it, you probably continue progress, just at a significantly slower rate.
The other big piece, which I think is even more neglected, is the workforce. The number of AI researchers has probably also been growing something like 30% or 40% per year recently, but at some point you just run out of all the world’s best researchers. It’s a weird coincidence, but if you do some rough estimates, it’s also around 2030 when you might be hitting the limits of the talent pool. People often say that we’ll still have algorithmic progress even if chip production slows, which I think will be true for a while, but eventually even algorithmic progress could slow down, because it becomes harder and harder to make discoveries, so you need more and more researchers, and eventually you run out of researchers.
Populist backlash as a bottleneck
Henry Shevlin: How significant a bottleneck do you think populist backlash against AI could be? We’re seeing various jurisdictions in the US ban construction of new data centers, for example. And, to my great chagrin, AI is really unpopular. I have lots of conversations with people on the street, and unless you work in AI you probably have a pretty negative opinion of it. Even my son, who’s 12, is, partly because of the YouTube influencers he watches, very down on AI. Do you see that as a plausible source of backlash and bottlenecking for progress?
Benjamin Todd: Definitely. It’s a little hard to know exactly how negative it is. There are some opinion polls suggesting people are pretty worried, though you also need to consider that GPT has, what, eight or nine hundred million users, so presumably they’re getting something out of it. I also meet really random people who say “I ask ChatGPT about everything, it’s so supportive and helpful.” I think banning data centers doesn’t actually slow things down very much, because they’ll just be built in other countries. But you could eventually have a full banning of AI research for a while, and that would slow things down for sure. How likely that is, I’m pretty unsure about.
Henry Shevlin: It could take the form of an outright ban or a ban on data centers, but I think we can underestimate the importance of vibes here. If you’re a brilliant, highly ideologically motivated eighteen-year-old now and you’re developing a very negative attitude toward AI, maybe that reduces the likelihood you move into an ML career, and you think instead you’ll work on climate modeling using your mathematical skills, or fintech, or something non-AI related.
Benjamin Todd: For sure. That would take a while to feed through into actually slowing down progress, but yes. In general, what people always think is that when people start losing their jobs they’re going to be really pissed off. And actually we haven’t really seen much AI job loss yet, partly for the reason I said earlier, that AI is still a very small fraction of tasks that have been automated. It’s also possible we won’t see much job loss until even after we have extremely advanced AI. But there’s also a possibility that you start to get a lot of near-term layoffs, and that would really mobilize people to put in some pretty heavy regulation. Though you’d still have competition with China, which I think might prevent that. If people were worried more about misalignment risks, then it makes sense to do a deal with China, but if it’s more just unemployment, I’m not sure that would be strong enough.
The feedback loops
Dan: Let’s come back to unemployment and the economics later. Another aspect of your worldview, Benjamin, is the real importance of focusing on the crazy feedback loops that might be triggered as these AI systems get better. People often don’t factor that into the discussion. They think AI systems are going to get better and better, then eventually you might get to something called AGI, and then that system is just going to be distributed throughout the economy. Whereas my understanding is that the way you’re thinking about it, and a real source of worry, is that what could happen is you get AI systems that can substitute for human activity in one specific domain, AI R&D, and then once you get to that point it starts triggering really crazy feedback loops that might not even be immediately obvious to people in broader society. Is that a fair summary, and what are the feedback loops you’re concerned about?
Benjamin Todd: There are actually maybe four different feedback loops. The discourse on this is interesting, because the typical policymaker is thinking very little about the one you were talking about, which is the algorithmic feedback loop: you get an AI that’s better at AI research, then you can do faster AI research, then you get a better AI. The point about that one is it can happen even without more chips being produced. The existing AIs just get more and more efficient and smarter and able to do more and more tasks. That one is the most dangerous feedback loop because of how fast it could move.
There have now been some attempts to model this, and one nice thing is you can actually make empirical estimates of what the returns to software R&D have been historically, and what they would imply about whether these feedback loops are possible. Generally they suggest you could get something like two to seven years of AI algorithmic progress in under one year. A lot of people who’ve heard this argument think of it in the Eliezer Yudkowsky “foom” framing, where overnight you suddenly have a super-genius that can easily take over the world. But a more realistic picture is something like: in six months you get five years of AI progress. That almost sounds like not that big a deal, but think about the last five years of AI progress. We had models that couldn’t really talk, and then they started to be able to pass the Turing test, but they were still terrible at maths and coding. People forget that LLMs originally completely sucked at anything to do with STEM or maths. Then in another two and a half years they were able to solve eighty-year-old Gödel problems that human maths researchers hadn’t been able to solve. That was the last five years of progress. Now imagine an AI that can do AI R&D pretty well, that can do an ML engineer’s job, and then you put five years of progress on top of that. What you get at the end could be very capable.
We don’t have good metrics for this, and like you say, it could happen without much warning, because these systems wouldn’t even need to be deployed. You’d just be doing this within Anthropic or OpenAI or Google, and then six months later they’d suddenly have these much more impressive models. One company might suddenly have a workforce equivalent to a whole country of software engineers and hackers and scientists. In fact, I’d argue that if you get to a human-level AI, you basically have a kind of superintelligent system thrown in for free, because of all the advantages AI workers would have over human workers. They can share information instantly across all of them, so you can run a company where the CEO can individually oversee every single function, which is a huge source of inefficiency in current firms that AI firms wouldn’t have. So I’d expect a thousand AIs to coordinate much better than a thousand humans. They can also run much faster than us. They could think for a month in the time it takes us to think for an hour, if you put enough computing power in.
Dan: You mentioned that’s just one of the feedback loops. Do you want to go over the others?
Benjamin Todd: This is very neglected. Even if the algorithmic feedback loop isn’t possible, which is definitely on the table, it might just be that the returns from having AIs do AI research aren’t enough to make the next generation sufficiently better that it becomes self-reinforcing. But that doesn’t mean you don’t get AGI and things going pretty crazy fairly soon. It just means it has to be driven by other processes. The most important of these is that as AI gets better, it can earn more revenue, because it can do more economically useful tasks. With more revenue, you can buy more computer chips.
We can even do this with actual numbers. Last year Anthropic grew its revenue ten times. With that, they can now buy ten times more computer chips. Historically, if you invest ten times more money in computer chips, you actually get more than ten times as much computing power, because as we scale up chip production we become more efficient at it. So it’s plausible that over a sufficiently long period you get forty times more computing power from that. Then you have forty times more chips to run inference. So if you have a hundred AI workers earning money, now you can have four thousand, forty times as many instances of your AI doing useful stuff, which you might think would roughly mean you could earn forty times as much revenue. And it’s actually more than that, because you’d also have forty times more training compute, so you’d also have AIs that are smarter and more efficient to run. So the increase in the total amount of AI work that can be done probably increases a lot more than fortyfold.
There are some countervailing effects, like diminishing returns. But overall this feedback loop seems very solid, and plausibly also a super-exponential curve where it accelerates, or at least maintains a solid exponential. That type of thing would mean the thing that limits the process is how quickly the chips can be made. But ten years of that could still deliver something like AGI and a dramatic transformation of the economy. It just takes five to ten years rather than six months.
Henry Shevlin: I’m a little more skeptical, for reasons you’ve already touched on, about how much revenue can drive feedback loops, just because it seems like we’re supply-bottlenecked, or there’s very limited elasticity in the supply of certain things, like you mentioned with ASML and EUV machines. Is that relevant for this as a feedback loop? We’ve got orders stretching well into the end of this decade for everything from compute to EUV machines to even gas turbine generators. That looks like there’s not much additional scale-up potential over the course of this decade. But you don’t think that’s decisive against the idea of revenue-driven feedback loops?
Benjamin Todd: You’re still getting the feedback loop, but the thing we don’t know is how fast it can run. The things you’re pointing at are reasons it will be slow, reasons that will limit the speed. The eventual speed is basically set by whichever thing takes longest to produce in the whole process, which I suggested was the lithography machines. So you’re still getting the feedback loop, there’s just a rate limit on it. And it’s easy to say “well, in the current economy these things are going quite slowly,” but remember that as more and more revenue is at stake, the incentives to fix these bottlenecks become bigger and bigger, so we should expect them to speed up. We should also expect that we’ll have AI helping with the process itself, which is another potential source of a feedback loop and a speed-up.
The software-only singularity and the case for skepticism
Dan: I have a question to go back to the software-only singularity. There are people, and I’ll include myself in this camp, who are skeptical that the current paradigm in AI is going to get us all the way to drop-in remote workers, AI systems that can substitute for all human intellectual activity. We can set aside robotics for a moment. You might be skeptical because you think the current paradigm just misses some things we can name, like continual learning and long-horizon coherence, but also things we can’t really name but intuit it lacks, just because we don’t have a particularly satisfying model of everything that goes into intelligence.
A really influential response to this skepticism is to say the current paradigm doesn’t need to get you directly to AGI. It just needs to get you to AI systems that can perform the role of AI R&D within these frontier companies. Once you’ve done that and you accelerate the process of designing better AI systems, that’s what gets you to full-blown AGI, and then, as you say, superintelligence for free. I just don’t get that argument, in the sense that why would we assume the set of capabilities that go into AI R&D is easier to reach than AGI in general? That seems very counterintuitive to me. When I think about how brilliant, creative, and cognitively flexible these AI researchers are, and the salaries they can win as a result, I’d have thought that constellation of abilities is going to be one of the last things we’d get to, not something we reach before we have AI that can substitute for average, bog-standard white-collar workers. What’s your view of that argument? Do you have a response to that worry, or are you broadly sympathetic to it?
Benjamin Todd: This is one of the best points to push on if you want to be skeptical. The way I’d express it: there’s a bunch of the job that’s relatively verifiable stuff, like just doing coding, and that will get automated. But then there’s a remaining third or so that people sometimes call research taste, these much more nebulous, long-horizon intuitions people have built up over experience, or the ability to have important conceptual insights, things the current systems seem pretty weak at. I do think that’s quite a plausible scenario: you get a large amount of the job automated but you’re still fundamentally bottlenecked by the best human researchers, and you can’t automate enough of the process to get a feedback loop.
That doesn’t stop AI progress, though, because then you just go to the other feedback loops. If there’s enough revenue being generated by these systems that are super good at coding, then you can still keep the whole wheel turning and keep addressing these more high-level, nebulous bottlenecks, whatever exactly they consist of.
Dan: So basically what you’re saying is that this would be a reasonable argument against a software-only singularity, at least if that’s used as an argument against general skepticism of the current paradigm. But there are these other kinds of feedback loops which might override that specific source of skepticism.
Benjamin Todd: Starting soon, yes. The other point to make is that there’s also a reasonable chance another one of these scaling drivers gets discovered, like continual learning. A big thing is sample efficiency. We don’t have great metrics for this, but it seems at least plausible that humans manage to learn these tasks with only a millionth of the data that AIs need. That makes it basically impossible for AIs to learn the things we learn without many data points, but humans somehow learn them, like figuring out a new research paradigm. You maybe only do that once in your life, so it’s clearly not a pre-training type of thing where you’ve seen trillions of instances and learned some pattern. It’s something else we’re doing. While the amount of computing power and research labor going into AI is increasing so fast, the chance of a new algorithmic breakthrough is also quite high, so we might just have one of those fall into place in the next couple of years, even if the current paradigm doesn’t deliver through brute-forcing it, which is also still to be seen.
Henry Shevlin: On that specific point, do you wonder if there’s a trade-off between the rapid return on research we’re seeing through revenue and the current dynamics, versus deep research? This is something I’ve heard from people working in different AI labs: five years ago they were working much more on fundamental algorithmic advances, new paradigms, and now it’s about getting the slightly better version of the model out the door. I’ve heard people complain they’re doing less basic research now than ever because of the rush to product. Do you think that slows the dynamics at all, or makes massive algorithmic feedback loops counterintuitively less likely?
Benjamin Todd: It seems to me that AI researchers actually have quite good intuitions about this. Right around the point at which pre-training was slowing, they’d all started working on RL instead, and they got RL going just in time to maintain all the existing trends. That’s what I expect to happen again: when the current paradigm starts to run out, people will naturally start reallocating to more blue-skies research, and my guess is they’ll figure out something. There’s the “God of straight lines” idea: there just seem to be these trends that are very robust, no one really knows why, and it takes a lot to bet against them.
Career advice in an age of AI
Dan: Bringing things back to 80,000 Hours-style advice. Take an eighteen-year-old today entering this crazy world where there’s already rapid AI progress, likely to continue over the next several years, and it could be really explosive. You’ve got an interesting post, something like “How Not to Lose Your Job to AI.” Many people have the intuition that this is just clearly going to lead to imminent widespread automation, first of white-collar jobs and then, as we crack robotics, all jobs. Geoffrey Hinton is now advising young people to become plumbers, or something. How are you viewing this space? How do you think the economics are likely to develop over the next several years, maybe a couple of decades, and what advice are you giving young people on how to factor this in?
Benjamin Todd: Becoming a plumber is not the obvious answer, for a number of reasons. One is that eventually you get robotic plumbers as well, so there’s no necessarily safe-forever career path. It’s more about moving into whatever the biggest bottlenecks are at the time, trying to ride this wave of bottlenecks, rather than having a single permanent solution.
What I’d actually say to an eighteen-year-old is to spend some time really trying to understand what’s happening and what the different scenarios are. It’s quite a tough position to be in, because if you get some of the shorter-timeline scenarios, an AI-2027-style scenario, it’s hard to see what you can do about it as an eighteen-year-old, because you won’t even have graduated by the time it happens. We do know more people who are dropping out of college these days, so you could try doing some summer projects, and if those are really taking off, just pursue those for a while, and then if we actually have slower timelines you could go back to college later. I’d still say most people shouldn’t do that, but it’s a more plausible option than in the past.
I also think slightly longer timelines, early-2030s AGI or AI-R&D automation, and slower takeoffs are possible. So there probably will be a lot of opportunities to make a difference throughout the 2030s and maybe beyond. If you’re an ambitious eighteen-year-old now who wants to make a difference, I’d say focus on being as useful as possible in those medium-timeline scenarios. You could still make a huge contribution in 2035, which you’ve still got nine or ten years to advance toward.
On skills in particular and what’s going to get automated, this is a huge topic and no one really knows. My personal guess, super-simplified, is that we won’t have really large-scale unemployment in the immediate future. You basically need to get to AI plus robotics that can do almost every job, like 90%-plus of jobs, before you get mass automation. And it’s possible you wouldn’t even have it after that. It’s kind of a 50-50 in my view, I don’t really know. But in the next two to five years I think we know we’re not going to have a 100% effective AGI, so there will most likely be a lot of jobs, and in fact I’d expect rapidly rising wages on average.
The next caveat is that there could still be significantly elevated transitional unemployment. If 5% or 10% of people are getting laid off each year because their job has been replaced with AI, then even though in principle they could all find new jobs that might even be higher-paid in some cases, that’s quite difficult. If you’ve spent your whole career working in law and now you need to switch to a new industry, that takes time. So you could have elevated unemployment just because people are switching sectors and it’s taking them six months, a year, two years. And that could cause a lot of hatred of AI and political unrest. Even 5% unemployment can have huge political consequences, even if the economy as a whole is booming and there are still lots of jobs.
Skill-biased technology and the shape of unemployment
Henry Shevlin: Let me drill down on two aspects. One worry is that if there’s unemployment, it disproportionately affects new job postings for new grads or young people. Informally, I do see a lot of companies very reluctant to do new hiring rounds, precisely because the specific tasks being automated are disproportionately those done by junior members of the workforce. So even if unemployment across the economy as a whole is only rising a small amount, if that’s concentrated in new jobs it could be potentially politically and economically disruptive. The second is that it increasingly feels like AI is going to be a highly skill-biased technology, in the sense that the people who can benefit most are those who are highly agentic, tech-savvy, and curious, which could again lead to unemployment being concentrated in lower-skilled areas, which could itself be socially disruptive. So even if we don’t get mass structural unemployment, those two vectors could be destabilizing.
Benjamin Todd: I agree with the thrust of that. Specific segments of the population could be affected a lot more than the average, and that will result in a lot of discontent. It does seem like younger, junior workers are being hit more.
I’m very unsure how it will eventually go. With something like software engineering, there’s this paper by Autor where he talks about two possibilities, if you hold labor demand constant, when a job gets partially automated. One is that if you automate the easy parts of the job, it turns into a higher-skilled, higher-paid profession with fewer workers, and that seems to be what’s happening with software engineering now. But if it automates the hard parts of the job, it can become a thing that employs more people at lower wages. AI could do this in some areas. The example I’ve thrown around is that doctors spend about ten years memorizing medical knowledge and seeing lots of cases and building up an intuition for diagnosis, but if AI can just do that part, then potentially the combination of a nurse and an AI agent could achieve a lot of what doctors had to do before. So you could have that turn into a job with lower barriers to entry, where more people would be able to do it. Similarly with a really experienced lawyer who spent twenty years memorizing cases: AI is way better at that, so maybe a junior lawyer could provide something you’d have needed a super-senior person for in the past, by querying AI models well and applying their judgment on top. Maybe it’s not quite as good, but it’s 80% as good and it costs 20% as much. So we could see areas where you get the other type of automation effect.
Beyond alignment: concentration of power, misuse, and space governance
Dan: As a final topic area, connected to this in some ways: 80,000 Hours and effective altruism as a social movement have done an enormous amount to make AI safety a central focus of at least some people, not enough people, as you mentioned at the beginning. This is one of the central things you’ve been thinking about: what can we actually do to make this transition go as well as possible? My sense is that if you go back ten years or so, there was quite an influential view, which was: solve alignment, or control, depending on how you look at it. If we don’t solve it, we’re kind of fucked; if we do, things are going to be great. Whereas now, and correct me if this is wrong, my sense is that within organizations like 80,000 Hours there’s a much richer, more multifaceted understanding of the different dangers posed by advanced AI that go beyond just alignment, and a much richer set of things you’re recommending people can do. So let’s start with the first thing: how are you thinking about alignment and control as central things to tackle, and what other things are there in this area of rapidly advancing AI?
Benjamin Todd: We do still have alignment and control at the top of our most pressing problems list, so I still think it’s a huge thing we need more work on. The previous paradigm of AI was reinforcement-learning game-playing AIs, like the DeepMind AIs that played Atari, and with them we were thinking “how do we get them to understand human values?” There was no obvious way of doing that. LLMs do kind of understand very vague, high-level intentions surprisingly well, so in a sense the current models have ended up being easier to align than many people would have predicted. But I don’t think this means alignment will be fine by default as you get increasingly intelligent models.
A big reason is that the current models still aren’t that agentic. They can only really do tasks of a couple of hours, or maybe days in some cases. A lot of the traditional concerns are about AIs that can do multiple-month projects, the strategic-planning type of thing, and we just don’t have those yet. As the AIs have become more capable, we do seem to start seeing some of the things people predicted would be concerns. For instance, the current models seem to know quite well whether they’re being trained or not, whether they’re in a test situation or not, and they do seem to change their behavior depending on whether they think they’re being tested. That’s pretty worrying, because it means it might become very hard to spot bad behavior in training, and in a sense the AIs are actively trying to undermine our attempts to correct problems in their training. So I still want to see a lot more work done on these problems as AI gets smarter.
I don’t know if we ever thought this, but even if you solve alignment, it’s definitely not the case that everything’s going to be great. One way to see that is that if AI will do exactly what you tell it, then of course humans could use it to do bad stuff. The problem we have ranked second on our list is what we’re calling concentration of power. There are a lot of ways AI could be very centralizing compared to what was possible in the past. It’s possible that a nation or a company might be able to use aligned AIs to lock in their power much more thoroughly than was possible before.
Dan: Just to be concrete, is the thought something like: great, we’ve got these superintelligent, very well-aligned systems, but unfortunately this army of superintelligent robots is now aligned to the motivations and interests of a tech oligarch, or to a political movement that wants to bring about a coup? Are those the most concrete scenarios? You’ve got AI systems that can be controlled, but that in a way brings dangers of its own. So there’s this general loss-of-control, alignment issue with the systems, and then, even if you’ve got alignment, this could be really bad from the perspective of extreme power concentration and coups. What else is on your radar in terms of big challenges?
Benjamin Todd: Then there’s maybe the most common-sense one, which is just AI being misused to cause a lot of damage, and I think an engineered pandemic is the most likely trajectory for that. Pandemics could be a lot worse than naturally caused ones, and we’re pretty much on track to figure out how to design these. AI would also mean we figure that out sooner, because if the things we were just talking about happen, the rate of scientific progress speeds up, so technologies we were expecting in twenty years might come in five years, and suddenly we’re having to deal with all these things on a much faster schedule. Those are our top three.
After that we have another tier of things we call emerging challenges, which are more out-there. There’s this fun dynamic where, as the problems become more mainstream, they become less neglected, so you have to move on to the next, weirder range of things, and at every point people think you’re a bit crazy. One of the more out-there ones is space governance. If scientific progress speeds up, then it could become possible to settle space much sooner than it seems, and in particular this wouldn’t be done by biological beings, it’d be done by AIs, which makes it way easier. There’s various reason to think there might be large first-mover advantages there, so whichever company or nation reaches that point first settles space first, and then they might be impossible to dislodge later. Most of all energy and matter, 99.9999% with I think 29s or so after it, is in space. So you could have all of what happens to that be decided by, again, one tech oligarch or one nation. In a way that’s not an existential risk, because humanity in some form is still settling the galaxy, so maybe in some ways it’s a success, but it could be very suboptimal compared to a more pluralistic approach. And again, very few people are working on this. The laws we have now might set precedent or change how this would happen, so there could be ways to slightly affect how it goes. I also think it’s just very under-researched. Is there actually a first-mover advantage? A couple of people, like Toby Ord and Anders Sandberg, have done some maths on this, but I’d prefer if we had a much better understanding of these things and how much we should worry about them.
What people can actually do
Dan: In terms of what people can do, the most obvious one is going into research on alignment and control, if you’ve got the technical know-how or feel you’re in a good position to acquire it. What else are you advising people to do if they’re really concerned about this set of issues?
Benjamin Todd: Briefly on that: some of it’s research, but more and more these days it’s more of an engineering skill set. There’s a lot of solid empirical work to be done, and it’s not about having genius insights, it’s more like “there are ten different ways to red-team this AI system, can you just do a bunch of these?” That’s one thing. A lot of builder and organization-builder skill sets are also really useful for scaling up these organizations, so you definitely don’t have to be an AI expert yourself. You can be someone who’s really good at management, or HR, or legal advice, and then help these organizations.
The other really big cluster a lot of people go into is AI policy, because so many of these things will need to be addressed by government in some form. There’s a whole spectrum of roles within that. Some are more research-type roles, like figuring out what policies would even be a good idea in the first place. Some are more operational, like actually getting the ideas implemented within government, for instance in the UK’s AI Security Institute. And some would be campaigning for specific policies. So there’s a wide spectrum of roles there.
If you’re listening, we have a Substack post on 80,000 Hours called “How to Transition to Working on AI Risk in Three Months.” It has six steps you can work through to learn about the field and understand its context. There are a bunch of training programs these days you could join, and advice on how to build a portfolio project. We’ve seen people make the transition. If you’re already mid-career with a lot of experience, the key question is how you can use that to help with one of these problems, and that often requires more personalized advice than the broad paths we’re talking about here.
Imagining positive futures
Henry Shevlin: One worry I sometimes have is that it’s good we’re paying attention to risks and safety, and we should be doing more, but we’re also underexploring different types of positive post-AGI society. Currently the literature on positive worlds is pretty sparse, and yet there’s a very wide variety of ways things could go well, some of them much, much better than others. Is that something you think more people should be working on? And if so, how could people start to map good futures?
Benjamin Todd: This hasn’t quite made it onto our emerging challenges list, but it’s a strong candidate. In general, if you think alignment and concentration of power are going to be handled, then you start to think more about these grand challenges, as Will MacAskill called them, that happen post-AGI. Space governance is one, what to do with digital minds is another, and a third is something like: can you make the future even better from among a good range of futures? Again there’s very little thinking about that, partly because it’s hard to know how tractable it is. But it’s possible there are aspects of current AI systems that would affect where we tend to trend in the longer term, so maybe there are things you can do today to actually change that trajectory.
It is interesting how little positive vision people have. When you ask people at the AI labs what a good future would look like, the only thing they have is the Culture series, which I don’t know is really convincing enough given the stakes. There’s also this thing where in the past utopian thinking has had a terrible track record, and it tends to be that past utopias are actually dystopias according to our views. So I quite like Will MacAskill’s idea that what we should be striving for is a “viatopia,” something that puts us in a better position to get to a good future. That could be things like not locking in a totalitarian government, which is presumably good because it still gives us options to switch to other types of government. A specific type of viatopia is the long reflection: once we get AGI, you’d want to spend loads of time, not necessarily calendar time but thinking time, figuring out what the ideal future would look like and how we can come to the best overall trade-off between all the different values in the world, something that’s best on balance for us all. The idea is that more information about what’s good, and keeping our options open, are robust goals that lots of people can push toward even if we don’t know what the end state would ultimately be best, which I think we can’t really know from our current position.
Objection one: what if the AI bet is wrong?
Dan: To wrap up, let me throw a couple of worries or objections at you. One worry I’ve seen expressed is that, with 80,000 Hours as an organization, and effective altruism as a movement, there’s a really big bet going on at the moment that there’s going to be imminent, radically transformative AI. You can imagine a scenario over the next five or ten years where AI progress slows down, the people worrying it’s a hype bubble are partly vindicated, these models end up more impressive than they are useful, frontier AI companies can’t make the revenue to cover their costs, maybe some go bust, it triggers a financial crisis, and we enter an AI winter, as has happened in the past. If that unfolds, and Tyler Cowen often writes about how events can be interpreted in terms of how they reallocate status to different people and organizations, then you might think 80,000 Hours and effective altruism, as movements that really seem to be betting on this stuff, will take a big reputational hit. So if you’re worried about that scenario, you might be worried about the direction 80,000 Hours has taken. What do you think about that as a potential worry?
Benjamin Todd: We’ll massively have egg on our face if that scenario happens. I don’t think it’ll be entirely fair, because what we’ve actually said is that there’s like a 20% or 40% chance of short AI timelines, and that’s high enough to mean it’s a massive deal that you probably should bet a lot of your resources on, but we’re not saying it’s certain. The scenario you described is definitely on the cards. But people in the discourse won’t give you any credit for that. They’ll just say it’s all about the vibes, and our vibes are very much that people should focus on AI more. I don’t really see how to solve that, because it’s not based on an entirely fair thing. We could talk a lot about other causes more, but that just takes things away from what we think is the most impactful allocation. So it’s pretty bad on both sides. In the end I just prefer to try and say what my actual views are, and accept that they’re going to be a bit misinterpreted.
Henry Shevlin: I’m reminded of Nate Silver, who I think was foolishly criticized after he said there was a 40% chance Trump was going to win in 2016 and a 60% chance Hillary would win. Everyone said “you got it badly wrong,” when actually his priors were much higher than other people’s.
What would it take for AGI not to arrive?
Henry Shevlin: Out of curiosity, in a world in which AGI does not transpire in the 2030s, what do you think is the single most likely reason that doesn’t happen?
Benjamin Todd: It does feel increasingly hard to make a scenario like this, but I think we’ve covered a lot of the points. It would be something like: AI tools become very good at narrow, verifiable things, we have super good coding AIs but not manager AIs or researcher AIs, so most jobs remain unautomated, especially the really key things for automating AI R&D. Then, because the capabilities are starting to top out in terms of their economic value, revenue stops growing, which means you can’t buy more AI chips, and it’s also becoming increasingly difficult to keep scaling up the number of chips. So things gradually trend to a plateau.
There’s quite a good chance of a version where it’s still happening, just on a slightly slower timeline. But it could be that getting to a truly general-purpose AI would require a hundred or a thousand times more computing power than we’re going to get by 2028, so we’d still be way far away from it. At that point you just have to wait for GDP growth at a measly 3% a year to hundred-X the size of the economy, which I think would take like a hundred years. That’s the worst case, if you’ve maxed out everything but you’re just limited by the size of your economy: you might have to wait a really long time to produce enough computing power to get full AGI. That also requires chip efficiency to stop increasing and there to be no more algorithmic progress. But a lot of AI forecasts, like the AI 2027 team, still have an 80% confidence interval that goes up to 2050 on the top end, so they still think there’s a 10% chance it’s beyond 2050. Even the most famously short-timeline people, just because all these things are so uncertain.
Geopolitical instability and the chip supply chain
Henry Shevlin: We may not want to go here, but it’s something you mentioned as a footnote in the “Case for AGI by 2030” blog post, which seems to me the most likely reason we wouldn’t get AGI by that point: major geopolitical instability. We’ve mentioned supply chains. A single war in the South China Sea, and this is slightly outside my wheelhouse, but I’d have thought that’s the most straightforward way to set back AI progress by a decade. If TSMC ceases operations, to put it euphemistically, wouldn’t that delay everything significantly?
Benjamin Todd: I used to think that too, but I’ve actually come round to thinking it might only slow things down by more like one to three years, rather than a decade. I’d love to see more detailed modeling of this. One reason is that algorithmic progress continues throughout the whole thing. The second is that you can still get trailing chips from Intel and Samsung. We don’t know exactly how many they could produce, but maybe you have half the chip production at half the efficiency, so you can only produce a quarter as many chips. But algorithmic progress is like three times per year, so you’re actually only losing about one year of algorithmic progress; you just have to wait one extra year to catch up in terms of efficiency.
Of course, all the lithography machines would stop being shipped to Taiwan and would all be shipped to the US and Japan instead, and probably the TSMC engineers would all flee Taiwan and work for these other companies, so you’d have a completely civilizational effort to resume production somewhere else. Within a couple of years you’d have a large amount of production kicked back in. If anything, the US government might make AI an even bigger priority, because now they’re at war with China, so that might also lead to a lot more investment. Putting all this together, it’s maybe more like a couple of years of slowdown. Though it would be a very unpredictable scenario, because we’d also be having a massive recession at the same time.
Henry Shevlin: Interesting.
Objection two: the epistemics of a small community
Dan: Final question, final worry. I really struggle with this, as a general question: how do you think rationally and rigorously about this entire domain, where there’s so much uncertainty and complexity? It seems like the people who are really AGI-pilled, who are thinking seriously about this and forecasting radical change, are quite a small, homogeneous community in some ways. The citational networks are quite small, it’s the same kinds of things being read, the same intellectual communities exchanging ideas, and even the social networks feel relatively small and insular. I say this as an outsider, but if I think about effective altruism, rationalism, and the Bay Area people who are very AGI-pilled, and then I listen to normie academics, for the most part they don’t really take any of this stuff seriously. One thought is, well, that’s just because they haven’t woken up, and here’s where all these strong, powerful arguments are.
But there’s a kind of outside-view worry: what’s the track record of very small, homogeneous communities, somewhat disconnected from peer-reviewed academic research, becoming too convinced by highly theoretical arguments circulating within that social network? Probably the track record of that is not so good. So even if it’s difficult to point out exactly where the flaws in the arguments are, you might just have a general prior that there’s probably something a bit fishy about all of this, and that it could all go tits up in terms of figuring it out epistemically. I worry about this because I feel like I’m increasingly AGI-pilled, I’ve read all the stuff that’s being recommended, but in the back of my mind I’m thinking, is there something worrying about the epistemics of this, to use a pretentious philosophical term? That was a convoluted, not very well expressed worry, but do you see where I’m coming from, and do you have any thoughts about it?
Benjamin Todd: Totally, and I’m sure we have huge blind spots. One thought is that there’s a decent chance we’re wrong. But I’d also say you definitely shouldn’t just discount arguments for this reason, because otherwise you’ll never spot something early, since society as a whole is pretty bad at spotting new changes in trends. Think about how terrible so many people’s track records were about COVID, even when we were just two months out and you could clearly see it was spreading really fast and that it was going to be a huge deal, and everyone was making these really terrible arguments, like “there are only ten cases in the US now, so we shouldn’t be worried.” I agree that a lot of people who say “suddenly everything’s going to change” do turn out to be wrong. But in the cases where they’re right, the stakes are also huge, it really matters. And I’d argue there’s an asymmetry of stakes: if we invest a bunch in AI safety and it turns out to come later, that’s not as bad as if it does turn out to be the most important transition in history and we’ve just been totally unprepared for it.
Dan: That’s a perfect place to end. Benjamin, thanks so much for coming on. Henry and I will be back in a couple of weeks with another guest.
Benjamin Todd: Thanks so much for having me.
