Watch now | Explosive technological progress, the "grand challenges" on the path to superintelligence, space expansion, and the ethics and politics of artificial minds
I wrote this long comment after listening to your podcast on AI existential risk with Ian Leslie (first part), and then realised that there is just a link to Leslie's Substack, which lets only his paid subscribers post comments. Maybe the next best place is here?
Fun fact that got me started, I quoted Leslie in your comment section in 2024: 'If ever there was a case that an information democracy, even a very messy one, is preferable to an information oligarchy, then the history of nutrition advice is it.' From his article "The Sugar Conspiracy", 2016. My worry was/is that your stance on epistemic gatekeeping, while way more based than most others I've come across, might be slightly too favourable to the gatekeepers, as it is not informed, to my knowledge, by their most egregious failure over the past decades, nutrition science.
As for AI, which might kill us full stop, as opposed to merely kill us prematurely like official nutrition advice, I look forward to your long essay, and would wait for it except that some feedback right here might possibly end up useful for the essay:
First of all, the podcast was very good I thought. And not only compared to the dismal average standard of the AI-ruin-skeptics camp! To name one example, I was impressed by your fair and respectful thoughts on the "rationalists". Clearly given their history and characteristics it's never going to be the case that they just made simple mistakes.
As far as I can tell, some strong thinkers on LessWrong agree with your skepticism about LLMs. Contra many at the labs. Then again, the crazy-bullish predictions of the labs, and of AI 2027, keep coming true. And importantly, LLM agents are reaching a point where, despite their limitations, they can strongly accelerate research into successor technologies. You mention Eliezer Yudkowsky and Scott Alexander: the latter, talking about AI 2027 when it first came out, said something like that the skeptic's 50-year timeline isn't actually reassuring when 50 years become five years (in the capabilities section you say "smuggling in so many different assumptions", but the assumptions seem to be all about timelines --- replace "intelligence" with "capabilities" if you must). And the former, a relative LLM skeptic to my knowledge, recently tweeted, replying to optimism about current alignment research: "Why would any of that apply to the post-transformer non-LLM AI that Dario will whip Mythos 5.4 into building for him?"
... which, unfortunately, may mean that the unpredicted niceness, stemming from LLM pre-training and invoked by you as empirical evidence against high existential risk, will be lost. The classic theory would be back in full force. Not least Yudkowsky's "value is fragile", directly contra your point (based on LLMs I presume) that alignment does not have to be *perfect* to prevent ruin.
Steven Byrnes on LessWrong, whose work looks particularly impressive to me (but I'm not an expert and have hardly managed to scratch the surface), expects "ruthless sociopath" RL-agent "brain-like" AGI. Apparently a number of high-profile figures, like Yann LeCun, work on such AI. I got that from the following post, maybe useful for your forthcoming essay as a counter-view to be fortified against. It also includes a separation between two markedly different "doomer" camps. You may well know all this, of course, but just in case.
I also had another thought. Which is that AI might become a victim of its own success. The more effective AI becomes in generating a return of investment for enterprise customers and the more compute becomes commodified the less appetite there will be in investors to let the AI companies blow that compute on diminishing returns in training future AI. So we could end up with a sort of economically driven technical stagnation. And in the marketplace of compute I'm not convinced academic research is going to win.
Parking the AI consciousness question. My initial thoughts on this were:
1. Most research isn't that helpful to humanity including a lot of say philosophy, social sciences and even hard sciences - academia and research is at least part a social game with its own rules and payoffs but not necessarily that beneficial more widely - no one needs a scholarly comparison of Henry James's poetry and his fiction - and as a clinician 90% of the medical research I read seems pointless
2. I'm not clear why we would invest all that AI infrastructure in research any more than we invest that much in pure research now with humans. What's the economic incentive? Surely more compute is going to get channeled down the same routes of industry and finance and consumer products rather than epoch defining scientific innovation - if isn't cost effective now with humans why will it be when it is AI?
3. I can see areas where AI might be able to lead advances such as in coding and computing where ideas can be developed directly but anyone working in academic research knows good ideas are two a penny - you need the boring and expensive physical research programme to test them out - for every hundred ideas about what drug might be helpful for a disease you'll have less than one actual viable medicine and millions spent on clinical trials - I'm not very convinced by the great man/great idea theory of science - I think it is a lot more contingent - you're only a visionary in retrospect
4. The discussion is very like the sort of EA and rationalist extrapolations to infinity you see in AI doomerism. Weirdly disembodied and abstract. But we're extrapolating a trend that isn't well defined. We don't have a good metric of intelligence but we're predicting it will increase in some exponential fashion rather than with diminishing returns due to physical constraints. That robotics will just sort that all out. Without worrying about GPUs, energy, physical space, compute, per token pricing etc. In the same way trains or planes don't now travel at 1 million miles an hour just because there were early trends that suggested linear or exponential gains there are likely to be future hard limits, physical or otherwise.
5. Obviously it is early days but it is also striking how limited the impact of AI has actually been. How narrow the use cases are currently despite the promise. Will we have space factories or just consumer facing chatbots? I do actually have some doubts about how feasible the whole self improving AI thing will be. I can see scope for refining harnesses and other aspects of the code but I'm yet to be convinced it will either be financially sustainable or technologically feasible to tell LLMs to revise themselves.
As an aside on the consciousness question I'm not going to into it in any depth but in was struck by "if consciousness is an illusion, it’s super weird that basically everyone who thinks about consciousness ends up with the same illusion, the same impressions" - because I'm pretty sure that the shrimp are not saying that!
Super thoughtful and reasonable conversation. Still hasn't convinced me we should be any more concerned about an intelligence explosion than a nuclear winter, but I appreciated the clear thinking. That said, it's hard for me to understand why Fin holds his views about likelihood given what he's willing to concede to Dan here:
> it’s sort of difficult for me to see how you’re going to get the really rapid increase in the rate of — let’s say research progress, or in the expansion of the amount of R&D that we can do — if AI systems are merely augmenting and extending human capabilities. Like, I totally understand the argument that says once AI systems can just flexibly substitute for human beings — so that you can get AI systems in some sense creating their own kind of autonomous domain of research — then just the numbers, in terms of how many AI systems we can get and how we might improve their capabilities and so on, that could result in this huge increase in the amount of research that we’re doing. That I buy. But the idea that you’re gonna get that merely by their augmenting human capabilities — that seems a bit more controversial to me.
Fin's response seems to be that, absent special increases in its capabilities, AI's effect on research productivity will probably be more of a 1-off. But if he's willing to concede that, then why should we believe we'll see continued increases in the right kind of capabilities for accelerating scientific discovery, particularly given how much harder each one is to get than the last one? And then even if continued AI advances _do_ manage to continue improving the pace of scientific discovery in ways that aren't 1-off, why should we think it would be a foom-like thing ("explosion") rather than something much more gradual, like we've seen since the enlightenment? At the end of the day, don't you kind of have to believe in a tightening feedback loop between researcher productivity and AI capabilities in order to compress the timeline?
In the Forethought paper from last year, https://www.forethought.org/research/preparing-for-the-intelligence-explosion he and Will seem to basically think this would happen either if AI could get good enough to substitute for research labor such that we can just massively expand supply of cognitive inputs, or if AI could get good enough to trigger a software based feedback loop to accelerate its own continued progress. But the first scenario just basically stipulates AGI, and the second stipulates RSI — and like, sure, fine _once_ you've assumed the magical things into your scenario, we can imagine all kinds of crazy stuff happening as a consequence. But why think those initial conditions would be likely to obtain at all?
Hard not to be left thinking we should approach this like the asteroid case — have a vague sense of what we'd do to prepare (launch some nukes into space?), what the signals would be that the asteroid was coming, and then only start preparing in earnest once we actually saw an asteroid on the radar.
It feels like consciousness is an illusion the same way atoms are a quantum illusion, in a reductionist sense yes, but the system phenomenon is casually pontent and reasonably stable. If I had to guess, consciousness is a confluence of a complex brain, language leading to complex abstraction and theory of mind that wraps back into self conception.
There may be parallel but distinct phenomena coming out of computing substrates (AI).
I wrote this long comment after listening to your podcast on AI existential risk with Ian Leslie (first part), and then realised that there is just a link to Leslie's Substack, which lets only his paid subscribers post comments. Maybe the next best place is here?
Fun fact that got me started, I quoted Leslie in your comment section in 2024: 'If ever there was a case that an information democracy, even a very messy one, is preferable to an information oligarchy, then the history of nutrition advice is it.' From his article "The Sugar Conspiracy", 2016. My worry was/is that your stance on epistemic gatekeeping, while way more based than most others I've come across, might be slightly too favourable to the gatekeepers, as it is not informed, to my knowledge, by their most egregious failure over the past decades, nutrition science.
As for AI, which might kill us full stop, as opposed to merely kill us prematurely like official nutrition advice, I look forward to your long essay, and would wait for it except that some feedback right here might possibly end up useful for the essay:
First of all, the podcast was very good I thought. And not only compared to the dismal average standard of the AI-ruin-skeptics camp! To name one example, I was impressed by your fair and respectful thoughts on the "rationalists". Clearly given their history and characteristics it's never going to be the case that they just made simple mistakes.
As far as I can tell, some strong thinkers on LessWrong agree with your skepticism about LLMs. Contra many at the labs. Then again, the crazy-bullish predictions of the labs, and of AI 2027, keep coming true. And importantly, LLM agents are reaching a point where, despite their limitations, they can strongly accelerate research into successor technologies. You mention Eliezer Yudkowsky and Scott Alexander: the latter, talking about AI 2027 when it first came out, said something like that the skeptic's 50-year timeline isn't actually reassuring when 50 years become five years (in the capabilities section you say "smuggling in so many different assumptions", but the assumptions seem to be all about timelines --- replace "intelligence" with "capabilities" if you must). And the former, a relative LLM skeptic to my knowledge, recently tweeted, replying to optimism about current alignment research: "Why would any of that apply to the post-transformer non-LLM AI that Dario will whip Mythos 5.4 into building for him?"
... which, unfortunately, may mean that the unpredicted niceness, stemming from LLM pre-training and invoked by you as empirical evidence against high existential risk, will be lost. The classic theory would be back in full force. Not least Yudkowsky's "value is fragile", directly contra your point (based on LLMs I presume) that alignment does not have to be *perfect* to prevent ruin.
Steven Byrnes on LessWrong, whose work looks particularly impressive to me (but I'm not an expert and have hardly managed to scratch the surface), expects "ruthless sociopath" RL-agent "brain-like" AGI. Apparently a number of high-profile figures, like Yann LeCun, work on such AI. I got that from the following post, maybe useful for your forthcoming essay as a counter-view to be fortified against. It also includes a separation between two markedly different "doomer" camps. You may well know all this, of course, but just in case.
https://www.lesswrong.com/posts/ZJZZEuPFKeEdkrRyf/why-we-should-expect-ruthless-sociopath-asi
I also had another thought. Which is that AI might become a victim of its own success. The more effective AI becomes in generating a return of investment for enterprise customers and the more compute becomes commodified the less appetite there will be in investors to let the AI companies blow that compute on diminishing returns in training future AI. So we could end up with a sort of economically driven technical stagnation. And in the marketplace of compute I'm not convinced academic research is going to win.
Parking the AI consciousness question. My initial thoughts on this were:
1. Most research isn't that helpful to humanity including a lot of say philosophy, social sciences and even hard sciences - academia and research is at least part a social game with its own rules and payoffs but not necessarily that beneficial more widely - no one needs a scholarly comparison of Henry James's poetry and his fiction - and as a clinician 90% of the medical research I read seems pointless
2. I'm not clear why we would invest all that AI infrastructure in research any more than we invest that much in pure research now with humans. What's the economic incentive? Surely more compute is going to get channeled down the same routes of industry and finance and consumer products rather than epoch defining scientific innovation - if isn't cost effective now with humans why will it be when it is AI?
3. I can see areas where AI might be able to lead advances such as in coding and computing where ideas can be developed directly but anyone working in academic research knows good ideas are two a penny - you need the boring and expensive physical research programme to test them out - for every hundred ideas about what drug might be helpful for a disease you'll have less than one actual viable medicine and millions spent on clinical trials - I'm not very convinced by the great man/great idea theory of science - I think it is a lot more contingent - you're only a visionary in retrospect
4. The discussion is very like the sort of EA and rationalist extrapolations to infinity you see in AI doomerism. Weirdly disembodied and abstract. But we're extrapolating a trend that isn't well defined. We don't have a good metric of intelligence but we're predicting it will increase in some exponential fashion rather than with diminishing returns due to physical constraints. That robotics will just sort that all out. Without worrying about GPUs, energy, physical space, compute, per token pricing etc. In the same way trains or planes don't now travel at 1 million miles an hour just because there were early trends that suggested linear or exponential gains there are likely to be future hard limits, physical or otherwise.
5. Obviously it is early days but it is also striking how limited the impact of AI has actually been. How narrow the use cases are currently despite the promise. Will we have space factories or just consumer facing chatbots? I do actually have some doubts about how feasible the whole self improving AI thing will be. I can see scope for refining harnesses and other aspects of the code but I'm yet to be convinced it will either be financially sustainable or technologically feasible to tell LLMs to revise themselves.
As an aside on the consciousness question I'm not going to into it in any depth but in was struck by "if consciousness is an illusion, it’s super weird that basically everyone who thinks about consciousness ends up with the same illusion, the same impressions" - because I'm pretty sure that the shrimp are not saying that!
Thanks. Good thoughts
Super thoughtful and reasonable conversation. Still hasn't convinced me we should be any more concerned about an intelligence explosion than a nuclear winter, but I appreciated the clear thinking. That said, it's hard for me to understand why Fin holds his views about likelihood given what he's willing to concede to Dan here:
> it’s sort of difficult for me to see how you’re going to get the really rapid increase in the rate of — let’s say research progress, or in the expansion of the amount of R&D that we can do — if AI systems are merely augmenting and extending human capabilities. Like, I totally understand the argument that says once AI systems can just flexibly substitute for human beings — so that you can get AI systems in some sense creating their own kind of autonomous domain of research — then just the numbers, in terms of how many AI systems we can get and how we might improve their capabilities and so on, that could result in this huge increase in the amount of research that we’re doing. That I buy. But the idea that you’re gonna get that merely by their augmenting human capabilities — that seems a bit more controversial to me.
Fin's response seems to be that, absent special increases in its capabilities, AI's effect on research productivity will probably be more of a 1-off. But if he's willing to concede that, then why should we believe we'll see continued increases in the right kind of capabilities for accelerating scientific discovery, particularly given how much harder each one is to get than the last one? And then even if continued AI advances _do_ manage to continue improving the pace of scientific discovery in ways that aren't 1-off, why should we think it would be a foom-like thing ("explosion") rather than something much more gradual, like we've seen since the enlightenment? At the end of the day, don't you kind of have to believe in a tightening feedback loop between researcher productivity and AI capabilities in order to compress the timeline?
In the Forethought paper from last year, https://www.forethought.org/research/preparing-for-the-intelligence-explosion he and Will seem to basically think this would happen either if AI could get good enough to substitute for research labor such that we can just massively expand supply of cognitive inputs, or if AI could get good enough to trigger a software based feedback loop to accelerate its own continued progress. But the first scenario just basically stipulates AGI, and the second stipulates RSI — and like, sure, fine _once_ you've assumed the magical things into your scenario, we can imagine all kinds of crazy stuff happening as a consequence. But why think those initial conditions would be likely to obtain at all?
Hard not to be left thinking we should approach this like the asteroid case — have a vague sense of what we'd do to prepare (launch some nukes into space?), what the signals would be that the asteroid was coming, and then only start preparing in earnest once we actually saw an asteroid on the radar.
Thanks - good comment.
It feels like consciousness is an illusion the same way atoms are a quantum illusion, in a reductionist sense yes, but the system phenomenon is casually pontent and reasonably stable. If I had to guess, consciousness is a confluence of a complex brain, language leading to complex abstraction and theory of mind that wraps back into self conception.
There may be parallel but distinct phenomena coming out of computing substrates (AI).