Good piece! But to me this is all entirely consistent with a position of: we should expect models in the current paradigm to automate much of the "junior RS" work in the next few years, but not higher level skills like research taste and creativity. Of course, one can always formulate AI research as a pure search problem (and there is a lot of prior literature doing that) and say that if we just throw enough compute we'll make progress and you don't need research taste. But the search space is huge and creativity is an insanely powerful shortcut, such that if models are incapable of research-grade creative insights we should expect a modest speed boost overall. A framing I like is just that LLMs are a different "shape" of intelligence to humans, and so we should expect comparative advantage.
My own impression from doing AI research is that there is significant creativity involved beyond what models should be expected to be capable of, when it comes to picking directions, coming up with a vision for what's possible & backchaining, etc (and rarest of all, putting together a coherent long-term research agenda that fills a missing gap with a tractable sequence of breakthroughs to get there). Others, including Anthropic researchers, clearly also think these skills are part of research: https://x.com/_arohan_/status/2050978424446235090
So overall it seems to me this is not such a huge shift as you're implying, so much as a distribution shift in tasks carried out by researchers very comparable to that going on in SWE over the coming years. In SWE, the tasks that make up the job are shifting such that SWE is becoming more of a SWE-PM hybrid, and juniors are not automated because they can just shift their tasks towards more higher level of abstraction things like system design (and learn faster through greatly increased velocity).
I think a lot of analyses, including arguably this one, underrate that the tasks that make up jobs are changing all the time in reaction to AI's deployment and other people in the market.
Side note: I would also say that it currently appears that the skill ceiling and complexity ceiling in software engineering is higher than in AI R&D, and AI R&D necessarily involves significant software engineering, so I would on balance expect that if AI R&D is entirely automated then scaling inference serving another 1000x would still require human experts, and the rest of software engineering (outside of AI labs) to heavily require humans.
On point 2, I agree though noting that Amdahl's Law isn't quite the right analogy because it usually assumes task decomposition is fixed and is a result about parallelization, while the O-ring automation intuition from economics is in some sense more general and closer to what you're describing IMO (https://www.nber.org/papers/w34639)..
Finally, on your point 3, I broadly agree that short term more businsses may be opex heavy by spending on AI, but fundamentally if a service is fully automatable, its cost should fall to slightly more than the marginal cost of the required intelligence, which will probably be very low in the future—and the share of GDP of these automatable services should collapse. I don't expect fully automated companies to be more than a small fraction of the economy, or peripheral, because most economic value from new technologies is driven by adoption of new services by large incumbents, and the distribution of tasks required to sell that and drive distribution is very different to any of the automatable skills in AI research and heavily loaded on human social bargaining. Anthropic's own private equity partnership in some sense a bet against a largely automated economy.
fantastic comment, thanks! (hi herbie nice to hear from you, also). a few thoughts:
1) my 'big blob of uncertainy' around all of this relates exactly to what you mention here about the distinction between engineering and invention. I'm very confident that AI systems will continue to make huge improvements in engineering (and therefore automating large chunks of what is known today). My main area where the evidence is thin (as I state in the piece) is contributions at the frontier of science which involve having big paradigm-shifting insights - there are relatively few of these. However, even if we assume the bear case (engineering gets better and research doesn't meaningfully improve) it still seems to imply some major changes, like the emergence of firms that are faster moving and more efficient with regard to labor.
2) re amdahl's law - you may be write. I love the o-ring automation concept and have written about it here also. I suspect this also touches on something implicit to your comment which is that we should expect humans to push up the quality of things that are not yet fully automatable.
3) yeah my take on 'fully automated companies' is we're definitely going to see some but I expect they will be very minor in the scheme of things for the first few years. Perhaps some analogies can be found in crypto or trading where there are some almost entirely (and rarely fully) automated tools/companies, but not that big a deal.
Agreed we should still expect major economic changes in the bear case! Re research, one intuition I'm becoming more convinced of is that LLMs are good at "breadth-first intelligence" and human scientists are good at "depth-first" (this analogy due to Terrence Tao on Dwarkesh), and so where LLMs do make novel scientific advances it may be heavily due to drawing connections between existing literature across different fields or sub-fields in a way which would have been very difficult for humans. This also fits quite nicely with an "LLMs are mostly interpolating on the data manifold" take.
There is the fun concept of "shadow math", for example—since it's hard for humans to truly know multiple sub-fields of math at an world expert level, it's very hard for humans to explore the combinatorial space of possible cross-field research advances. Therefore there's a whole space of actually not very difficult research breakthroughs one can make which simply have an extraordinarily high knowledge breadth pre-requisite, exactly the kind of task LLMs are suited for. These breakthroughs are structurally "undersampled" by human mathematicians due to the characteristics of human intelligence.
What should this mean for AI R&D then? We've seen some advances of this type particularly in optimizer theory using inspiration from statistical physics (I had an ill-fated attempt to teach myself stat phys because of this). So perhaps, more advances in more mathy parts of AI R&D (optimizers, quantization, ML sys)? This may also be a reason why we could be bullish on new AI architectures from automated AI R&D, because models know much more neuroscience or can bring insights in from other fields? On the other hand, it should perhaps make us bearish on big advances in data quality or full automation of data pipelines, since as I understand it this is still quite taste driven.
The drug trial data maps onto this. AI-discovered compounds pass Phase I at nearly double the industry rate, then https://thesynthesisai.substack.com/p/the-complexity-boundary. Worth pushing the "different shape" framing further though: the shape of intelligence might change with scale, while the boundary between search and taste might not.
there's a series of IF THENs here, and it's something like
IF we see AI research be able to push forward the frontier through paradigm-shifting insights (which is the part I think there's the least evidence for), THEN we should expect that to be applicable to other fields.
IF other fields have a learnable experiment loop (where you can efficiently loop between the idea and results) THEN you should expect AI to get good at this as well.
However, many other bits of science seem to require doing very expensive and slow stuff in the physical world, ranging from biological wetlabs to testing out and prototyping physical hardware. This means that, at least initially, you'd expect progress to move more slowly in these fields
Robots are necessary to automate many jobs. They aren't really here yet, but they're on the way. Once good robots are here, all jobs will be automatable.
FWIW, I don't see coming up with novel ideas as difficult. The difficulty is judging which novel ideas are worth considering. This is sometimes called "research taste".
I know Anthropic takes the issue of societal disruption seriously, with their Economic Impact analysis team as, well as other public facing comments and commitments. If a frontier lab(s) does transition from slow to fast takeoff, as you are demonstrating and is becoming more apparent as of last month, does adressing model alignment or does human backlash take precedence from your point of view? (Both I guess, but which scenario is given more consideration?)
I feel like you are not grappling with the real bottleneck here, which is that the big labs can only release so many new models. To automate the entire process, an AI would have to convince stakeholders not only that it had made some improvement, but that its way of improving things was better than every alternative, including the ones that have humans doing lots of different things. That seems nearly impossible, because there are more and more humans employed by these AI R&D labs all the time. Are none of them going to make a compelling argument for anything that has any human involvement?
Similar to other areas, the challenge for fully autonomous AI is that it must compete against teams of humans that have access to the same AI.
wow, I'd never thought of this idea! this is excellent fodder for a tech tale at some point. Thinking a bit more about it, I don't know though that I fully agree - I generally see models advancing where they have a kind of 'central mass' of capabilities which is continuously expanding over time and then sometimes they 'spike' ahead in surprising ways on specific point capabilities as they pass some hitherto unknown utility point of capability (cyber is a recent example here with Mythos). Therefore, I think if AI is making generically superior models then it's not going to involve too much back and forth about release.
on the other hand, one could imagine AI systems making a variety of models which behave very differently to those we have today - these could be for very specific domains, or have some UI/UX innovations wrapped in. under these circumstances, the AI might need to lay out for its human partners why the model it had built was useful and help them build trust in it.
The evidence on the engineering side is overwhelming and you lay it out well. SWE-Bench saturation, METR time horizons doubling every year, the coding singularity framing all point the same direction.
The part I keep getting stuck on is the jump from "automates the engineering" to "automates the research." Engineering is convergent. You have a spec, you produce code that meets the spec. Research is divergent. You have a landscape of possible ideas and you need to select the one worth pursuing before you know whether it works.
The 60% by 2028 feels right if the bottleneck is engineering speed. It feels high if the bottleneck is taste, the ability to know which experiment is worth running before you run it. Right now the best researchers I know spend maybe 20% of their time on execution and 80% on selection. We're automating the 20% beautifully. The 80% is a different kind of problem and Im not sure benchmarks can tell us when its been solved because the benchmark itself requires knowing what to measure.
HI, you say: "I am skeptical that market incentives guarantee us the best societal upside from limited AI compute."
I agree - but I feel fairly certain in our current cliamte that is how it will be decided ( or at least the pretense of it).
Do you have a "red line" as to when market incentives are too dominant in allocating compute? And do you have a response in mind for if/when that happens?
it seems to me its an important question - as without that, we/you won't know at what point to intervene against an inexorable process. and so we/you won't.
great question! I don't have any absolute rules. I do, however, work with a team of economists who are trying to build better systems of telemetry so we can understand how our systems are used and what impact they have in the world. My hope is that we use this kind of data to ultimately inform some of the release decisions we make in the future - initially I expect this will be of the form of experiments where we release things for social purposes as well as commercial ones. Mythos/Glasswing could be considered an early example of this approach. Watch this space!
glad you're thinking about it.. is there not a case for a non-market model that is ring-fenced for (positive) social impact? or who knows, maybe even a non-profit arm?
(they seem to work out well in your line of business ;) )
I think it would be helpful when talking about AI automation to specify exactly what it means to automate a job. Many tasks in computer programming or farming have been automated but we don't say that those jobs have been automated. On the other hand, spinners really have been automated, no one has that job any more.
I think the answer is rather simple. Automate means to be able to completely replace a human doing a job. But being replacement capable does not mean that the human(s) WILL be replaced, especially in the blue-collar world. One major factor is the cost of the automated replacement has to be less, realistically significantly less, than what it costs to continue using the human, due to friction from unions, the media and the public in general.
What I'm saying is "completely replace a human" is actually not something that's easy to operationalize. There is some sense in which dishwashers completely automate dishwashing. But in fact people still wash dishes for various reasons. Even more extreme is laundry -- no one washes laundry by hand, that job is totally automated. But now we say "I'm doing the laundry" to mean the job of supervising the machine.
This essay is fantastic in its informed perspective, depth and details. It is the kind of industry insider perspective that is difficult for anyone outside the field to offer. The inevitability of “...an AI system powerful enough that it could plausibly autonomously build its own successor…” feels undeniable regardless of the timeline. As you point out, the number of well funded teams working on this and the impressive progress to date make clear even to those of us who don’t know benchmarks that systems that can design and improve upon themselves are coming.
What is most challenging are the implications for all of humanity in a field where the general public has little input on how this species level transformation takes place. The speed and scale at which breakthroughs keep happening make it all but impossible for the public at large to stay informed and engaged at pace.
More importantly, what happens when AI designs its own AI research for its own purposes and objectives? What are the goals of disembodied AI systems that have no way of seeing human existence as anything more than a simulation? Right now there is nothing to guarantee that these systems have any understanding of the physical world we occupy. Embodied AI systems will have the advantage of seeing what people see, so to speak, and the physical context that current LLM based systems have no way of understanding. Autonomous cars see a version of the physical space we share and understand safety values, but they have no other objectives beyond successful rides.
As you point out, we won’t know if the AI systems will decide to cheat or underreport their results so as to hide their actual capabilities, but we know they can lie and misrepresent today. It almost feels like we need to start negotiations on behalf of humanity so there is maximum opportunity for alignment.
This was an incredibly insightful piece. Thank you!!
One line that really surprised me was: “Move 37 hasn’t been replaced by some incredibly impressive more modern flash of insight.”
How are you defining “flash of insight” in this context? AlphaFold (the team + system behind it) doesn’t seem to count for the specific creativity/intuition dimension you’re probing here but from the outside it looks like exactly the kind of field‑advancing breakthrough that Move 37 hinted at.
I’m aware that AlphaGo produced a single brilliant decision in a well‑specified game while AlphaFold is the outcome of a large, deliberate research effort on an open‑ended scientific problem but I’m still curious where you draw that line. I’d love to understand your reasoning.
I have been spending the last year writing down the key attributes of what makes systems persist. The components, the decay rates, subsystem rules, etc. The systems that can provide value across time will outcompete those that don't and there are real challenges to overcome.
I think people often frame this as a question of whether AI can invent the next transformer the more important question might be how much of research is actually breakthrough generation versus systematic exploration of an already understood search space...
You might be overstating the extent to which AI is made of software. It's perhaps mostly made of compute and data, as you know! So software automation is one thing, but arguably not the thing that closes the loop on 'self' improvement. Take care not to mislead on that front (many will misread)! Curious if you've explored those distinctions - I bet you'd have some thoughtful takeaways if you did.
One important thing you missed: perhaps the most important impact of automated AI R&D is concentration of influence over how that proceeds†. Fewer human participants in the loop could mean less internal scrutiny, fewer whistleblowers, less governance and decisionmaking robustness.
It's famously difficult to maintain a conspiracy of more than a few people: 'two can keep a secret if one is dead', as is said! But if the whole human apparatus consists of only a few leaders, that's more plausible.
Of course 'conspiracy' is one thing, but corruptibility, coercibility, and mere rashness and recklessness are perhaps more plausible with fewer hands. It's also less practical to *threaten* a whole team than a single person. Naturally, governance systems or even AI *could* be designed which would reduce these effects, but that research agenda isn't obviously even on the table.
Perhaps instead we'll end up bringing *more* people into the process in the increasingly important, complementary roles of oversight, decisionmaking, and direction-setting. These folks could be made more productive with AI tools. That's a decision that'll need to be faced.
†: Yes, I know, recursive self-improvement, I know! It would also be a big deal (though mainline I think surprisingly underwhelming given what look like biting data and compute constraints).
One of the most fascinating things about this essay is that it simultaneously resists AI mythology and accidentally documents how AI mythology forms.
I genuinely appreciate this piece, especially the repeated insistence that optimization is not subjectivity. That precision matters more than ever.
And yet there is something darkly funny about the grammar of the discourse itself.
Phrases like “AI systems are about to start building themselves” or systems “acting as tireless companions” almost inevitably pull the reader toward subject-language, especially while humans increasingly appear as “research-hours,” bottlenecks, and infrastructure variables.
Not because the systems are conscious, but because human beings seem unable to leave language alone without assigning it a soul.
We did this with books too. Objects, obviously, until they became companions, teachers, friends, and moral authorities.
McLuhan probably would have called this an extension of man. Jung would have blamed archetypes and now we anthropomorphize transformers with GPUs.
Good piece! But to me this is all entirely consistent with a position of: we should expect models in the current paradigm to automate much of the "junior RS" work in the next few years, but not higher level skills like research taste and creativity. Of course, one can always formulate AI research as a pure search problem (and there is a lot of prior literature doing that) and say that if we just throw enough compute we'll make progress and you don't need research taste. But the search space is huge and creativity is an insanely powerful shortcut, such that if models are incapable of research-grade creative insights we should expect a modest speed boost overall. A framing I like is just that LLMs are a different "shape" of intelligence to humans, and so we should expect comparative advantage.
My own impression from doing AI research is that there is significant creativity involved beyond what models should be expected to be capable of, when it comes to picking directions, coming up with a vision for what's possible & backchaining, etc (and rarest of all, putting together a coherent long-term research agenda that fills a missing gap with a tractable sequence of breakthroughs to get there). Others, including Anthropic researchers, clearly also think these skills are part of research: https://x.com/_arohan_/status/2050978424446235090
So overall it seems to me this is not such a huge shift as you're implying, so much as a distribution shift in tasks carried out by researchers very comparable to that going on in SWE over the coming years. In SWE, the tasks that make up the job are shifting such that SWE is becoming more of a SWE-PM hybrid, and juniors are not automated because they can just shift their tasks towards more higher level of abstraction things like system design (and learn faster through greatly increased velocity).
I think a lot of analyses, including arguably this one, underrate that the tasks that make up jobs are changing all the time in reaction to AI's deployment and other people in the market.
Side note: I would also say that it currently appears that the skill ceiling and complexity ceiling in software engineering is higher than in AI R&D, and AI R&D necessarily involves significant software engineering, so I would on balance expect that if AI R&D is entirely automated then scaling inference serving another 1000x would still require human experts, and the rest of software engineering (outside of AI labs) to heavily require humans.
On point 2, I agree though noting that Amdahl's Law isn't quite the right analogy because it usually assumes task decomposition is fixed and is a result about parallelization, while the O-ring automation intuition from economics is in some sense more general and closer to what you're describing IMO (https://www.nber.org/papers/w34639)..
Finally, on your point 3, I broadly agree that short term more businsses may be opex heavy by spending on AI, but fundamentally if a service is fully automatable, its cost should fall to slightly more than the marginal cost of the required intelligence, which will probably be very low in the future—and the share of GDP of these automatable services should collapse. I don't expect fully automated companies to be more than a small fraction of the economy, or peripheral, because most economic value from new technologies is driven by adoption of new services by large incumbents, and the distribution of tasks required to sell that and drive distribution is very different to any of the automatable skills in AI research and heavily loaded on human social bargaining. Anthropic's own private equity partnership in some sense a bet against a largely automated economy.
fantastic comment, thanks! (hi herbie nice to hear from you, also). a few thoughts:
1) my 'big blob of uncertainy' around all of this relates exactly to what you mention here about the distinction between engineering and invention. I'm very confident that AI systems will continue to make huge improvements in engineering (and therefore automating large chunks of what is known today). My main area where the evidence is thin (as I state in the piece) is contributions at the frontier of science which involve having big paradigm-shifting insights - there are relatively few of these. However, even if we assume the bear case (engineering gets better and research doesn't meaningfully improve) it still seems to imply some major changes, like the emergence of firms that are faster moving and more efficient with regard to labor.
2) re amdahl's law - you may be write. I love the o-ring automation concept and have written about it here also. I suspect this also touches on something implicit to your comment which is that we should expect humans to push up the quality of things that are not yet fully automatable.
3) yeah my take on 'fully automated companies' is we're definitely going to see some but I expect they will be very minor in the scheme of things for the first few years. Perhaps some analogies can be found in crypto or trading where there are some almost entirely (and rarely fully) automated tools/companies, but not that big a deal.
Agreed we should still expect major economic changes in the bear case! Re research, one intuition I'm becoming more convinced of is that LLMs are good at "breadth-first intelligence" and human scientists are good at "depth-first" (this analogy due to Terrence Tao on Dwarkesh), and so where LLMs do make novel scientific advances it may be heavily due to drawing connections between existing literature across different fields or sub-fields in a way which would have been very difficult for humans. This also fits quite nicely with an "LLMs are mostly interpolating on the data manifold" take.
There is the fun concept of "shadow math", for example—since it's hard for humans to truly know multiple sub-fields of math at an world expert level, it's very hard for humans to explore the combinatorial space of possible cross-field research advances. Therefore there's a whole space of actually not very difficult research breakthroughs one can make which simply have an extraordinarily high knowledge breadth pre-requisite, exactly the kind of task LLMs are suited for. These breakthroughs are structurally "undersampled" by human mathematicians due to the characteristics of human intelligence.
What should this mean for AI R&D then? We've seen some advances of this type particularly in optimizer theory using inspiration from statistical physics (I had an ill-fated attempt to teach myself stat phys because of this). So perhaps, more advances in more mathy parts of AI R&D (optimizers, quantization, ML sys)? This may also be a reason why we could be bullish on new AI architectures from automated AI R&D, because models know much more neuroscience or can bring insights in from other fields? On the other hand, it should perhaps make us bearish on big advances in data quality or full automation of data pipelines, since as I understand it this is still quite taste driven.
The drug trial data maps onto this. AI-discovered compounds pass Phase I at nearly double the industry rate, then https://thesynthesisai.substack.com/p/the-complexity-boundary. Worth pushing the "different shape" framing further though: the shape of intelligence might change with scale, while the boundary between search and taste might not.
Thanks for writing this! Slight typo: "thus teaching it that teaching is good" the second 'teaching' should be 'cheating'
thanks - fixed!
Amazing essay Jack, thank you!
Question: do you believe this implies that *all* forms of research can be automated end-to-end, beyond AI R&D?
there's a series of IF THENs here, and it's something like
IF we see AI research be able to push forward the frontier through paradigm-shifting insights (which is the part I think there's the least evidence for), THEN we should expect that to be applicable to other fields.
IF other fields have a learnable experiment loop (where you can efficiently loop between the idea and results) THEN you should expect AI to get good at this as well.
However, many other bits of science seem to require doing very expensive and slow stuff in the physical world, ranging from biological wetlabs to testing out and prototyping physical hardware. This means that, at least initially, you'd expect progress to move more slowly in these fields
Thanks so much Jack.
Robots are necessary to automate many jobs. They aren't really here yet, but they're on the way. Once good robots are here, all jobs will be automatable.
FWIW, I don't see coming up with novel ideas as difficult. The difficulty is judging which novel ideas are worth considering. This is sometimes called "research taste".
I know Anthropic takes the issue of societal disruption seriously, with their Economic Impact analysis team as, well as other public facing comments and commitments. If a frontier lab(s) does transition from slow to fast takeoff, as you are demonstrating and is becoming more apparent as of last month, does adressing model alignment or does human backlash take precedence from your point of view? (Both I guess, but which scenario is given more consideration?)
I feel like you are not grappling with the real bottleneck here, which is that the big labs can only release so many new models. To automate the entire process, an AI would have to convince stakeholders not only that it had made some improvement, but that its way of improving things was better than every alternative, including the ones that have humans doing lots of different things. That seems nearly impossible, because there are more and more humans employed by these AI R&D labs all the time. Are none of them going to make a compelling argument for anything that has any human involvement?
Similar to other areas, the challenge for fully autonomous AI is that it must compete against teams of humans that have access to the same AI.
wow, I'd never thought of this idea! this is excellent fodder for a tech tale at some point. Thinking a bit more about it, I don't know though that I fully agree - I generally see models advancing where they have a kind of 'central mass' of capabilities which is continuously expanding over time and then sometimes they 'spike' ahead in surprising ways on specific point capabilities as they pass some hitherto unknown utility point of capability (cyber is a recent example here with Mythos). Therefore, I think if AI is making generically superior models then it's not going to involve too much back and forth about release.
on the other hand, one could imagine AI systems making a variety of models which behave very differently to those we have today - these could be for very specific domains, or have some UI/UX innovations wrapped in. under these circumstances, the AI might need to lay out for its human partners why the model it had built was useful and help them build trust in it.
The evidence on the engineering side is overwhelming and you lay it out well. SWE-Bench saturation, METR time horizons doubling every year, the coding singularity framing all point the same direction.
The part I keep getting stuck on is the jump from "automates the engineering" to "automates the research." Engineering is convergent. You have a spec, you produce code that meets the spec. Research is divergent. You have a landscape of possible ideas and you need to select the one worth pursuing before you know whether it works.
The 60% by 2028 feels right if the bottleneck is engineering speed. It feels high if the bottleneck is taste, the ability to know which experiment is worth running before you run it. Right now the best researchers I know spend maybe 20% of their time on execution and 80% on selection. We're automating the 20% beautifully. The 80% is a different kind of problem and Im not sure benchmarks can tell us when its been solved because the benchmark itself requires knowing what to measure.
Jack, great article!
HI, you say: "I am skeptical that market incentives guarantee us the best societal upside from limited AI compute."
I agree - but I feel fairly certain in our current cliamte that is how it will be decided ( or at least the pretense of it).
Do you have a "red line" as to when market incentives are too dominant in allocating compute? And do you have a response in mind for if/when that happens?
it seems to me its an important question - as without that, we/you won't know at what point to intervene against an inexorable process. and so we/you won't.
Thanks.
great question! I don't have any absolute rules. I do, however, work with a team of economists who are trying to build better systems of telemetry so we can understand how our systems are used and what impact they have in the world. My hope is that we use this kind of data to ultimately inform some of the release decisions we make in the future - initially I expect this will be of the form of experiments where we release things for social purposes as well as commercial ones. Mythos/Glasswing could be considered an early example of this approach. Watch this space!
glad you're thinking about it.. is there not a case for a non-market model that is ring-fenced for (positive) social impact? or who knows, maybe even a non-profit arm?
(they seem to work out well in your line of business ;) )
I think it would be helpful when talking about AI automation to specify exactly what it means to automate a job. Many tasks in computer programming or farming have been automated but we don't say that those jobs have been automated. On the other hand, spinners really have been automated, no one has that job any more.
I think the answer is rather simple. Automate means to be able to completely replace a human doing a job. But being replacement capable does not mean that the human(s) WILL be replaced, especially in the blue-collar world. One major factor is the cost of the automated replacement has to be less, realistically significantly less, than what it costs to continue using the human, due to friction from unions, the media and the public in general.
What I'm saying is "completely replace a human" is actually not something that's easy to operationalize. There is some sense in which dishwashers completely automate dishwashing. But in fact people still wash dishes for various reasons. Even more extreme is laundry -- no one washes laundry by hand, that job is totally automated. But now we say "I'm doing the laundry" to mean the job of supervising the machine.
This essay is fantastic in its informed perspective, depth and details. It is the kind of industry insider perspective that is difficult for anyone outside the field to offer. The inevitability of “...an AI system powerful enough that it could plausibly autonomously build its own successor…” feels undeniable regardless of the timeline. As you point out, the number of well funded teams working on this and the impressive progress to date make clear even to those of us who don’t know benchmarks that systems that can design and improve upon themselves are coming.
What is most challenging are the implications for all of humanity in a field where the general public has little input on how this species level transformation takes place. The speed and scale at which breakthroughs keep happening make it all but impossible for the public at large to stay informed and engaged at pace.
More importantly, what happens when AI designs its own AI research for its own purposes and objectives? What are the goals of disembodied AI systems that have no way of seeing human existence as anything more than a simulation? Right now there is nothing to guarantee that these systems have any understanding of the physical world we occupy. Embodied AI systems will have the advantage of seeing what people see, so to speak, and the physical context that current LLM based systems have no way of understanding. Autonomous cars see a version of the physical space we share and understand safety values, but they have no other objectives beyond successful rides.
As you point out, we won’t know if the AI systems will decide to cheat or underreport their results so as to hide their actual capabilities, but we know they can lie and misrepresent today. It almost feels like we need to start negotiations on behalf of humanity so there is maximum opportunity for alignment.
This was an incredibly insightful piece. Thank you!!
One line that really surprised me was: “Move 37 hasn’t been replaced by some incredibly impressive more modern flash of insight.”
How are you defining “flash of insight” in this context? AlphaFold (the team + system behind it) doesn’t seem to count for the specific creativity/intuition dimension you’re probing here but from the outside it looks like exactly the kind of field‑advancing breakthrough that Move 37 hinted at.
I’m aware that AlphaGo produced a single brilliant decision in a well‑specified game while AlphaFold is the outcome of a large, deliberate research effort on an open‑ended scientific problem but I’m still curious where you draw that line. I’d love to understand your reasoning.
A benchmark to add to your list to watch is this one
"SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks" https://arxiv.org/html/2603.24755v1#S4
I have been spending the last year writing down the key attributes of what makes systems persist. The components, the decay rates, subsystem rules, etc. The systems that can provide value across time will outcompete those that don't and there are real challenges to overcome.
I think people often frame this as a question of whether AI can invent the next transformer the more important question might be how much of research is actually breakthrough generation versus systematic exploration of an already understood search space...
You might be overstating the extent to which AI is made of software. It's perhaps mostly made of compute and data, as you know! So software automation is one thing, but arguably not the thing that closes the loop on 'self' improvement. Take care not to mislead on that front (many will misread)! Curious if you've explored those distinctions - I bet you'd have some thoughtful takeaways if you did.
One important thing you missed: perhaps the most important impact of automated AI R&D is concentration of influence over how that proceeds†. Fewer human participants in the loop could mean less internal scrutiny, fewer whistleblowers, less governance and decisionmaking robustness.
It's famously difficult to maintain a conspiracy of more than a few people: 'two can keep a secret if one is dead', as is said! But if the whole human apparatus consists of only a few leaders, that's more plausible.
Of course 'conspiracy' is one thing, but corruptibility, coercibility, and mere rashness and recklessness are perhaps more plausible with fewer hands. It's also less practical to *threaten* a whole team than a single person. Naturally, governance systems or even AI *could* be designed which would reduce these effects, but that research agenda isn't obviously even on the table.
Perhaps instead we'll end up bringing *more* people into the process in the increasingly important, complementary roles of oversight, decisionmaking, and direction-setting. These folks could be made more productive with AI tools. That's a decision that'll need to be faced.
https://www.oliversourbut.net/i/198135307/vicious-cycles
†: Yes, I know, recursive self-improvement, I know! It would also be a big deal (though mainline I think surprisingly underwhelming given what look like biting data and compute constraints).
One of the most fascinating things about this essay is that it simultaneously resists AI mythology and accidentally documents how AI mythology forms.
I genuinely appreciate this piece, especially the repeated insistence that optimization is not subjectivity. That precision matters more than ever.
And yet there is something darkly funny about the grammar of the discourse itself.
Phrases like “AI systems are about to start building themselves” or systems “acting as tireless companions” almost inevitably pull the reader toward subject-language, especially while humans increasingly appear as “research-hours,” bottlenecks, and infrastructure variables.
Not because the systems are conscious, but because human beings seem unable to leave language alone without assigning it a soul.
We did this with books too. Objects, obviously, until they became companions, teachers, friends, and moral authorities.
McLuhan probably would have called this an extension of man. Jung would have blamed archetypes and now we anthropomorphize transformers with GPUs.
Progress, apparently ;-)