This post reflects solely my own views and does not reflect the views of OpenAI or any of my colleagues at the company.
Last week, the AI safety non-profit Model Evaluation and Threat Research (METR) released a study on “AI R&D Acceleration” at Anthropic. The study, based on extensive access granted by Anthropic to METR staff, finds that, “as of October 2026, Anthropic is likely improving the capabilities of their models on well-specified tasks at around twice the rate they did between March 2024 and October 2025.” The METR finding came a few days after Anthropic, in its amended S-1 filing ahead of their planned initial public offering, disclosed that it “may imminently meet our Responsible Scaling Policy’s definition of ‘dramatically accelerated progress attributable to AI.’”
The METR study, combined with the earlier disclosure from Anthropic, provoked a cascade of reaction online. Some prominent members of the AI safety community went so far as to say that METR’s findings about Anthropic’s use of AI to accelerate AI research made them feel physically ill. Others, including my new OpenAI colleague David Holtz and Google DeepMind’s Seb Krier (both tweeting in their personal capacities and not on behalf of their respective employers), were more skeptical of the results, pointing out a lack of empirical rigor (due, in no small part, to Anthropic’s aggressive redactions of data and other information from METR’s public report) or observing that Anthropic’s capabilities growth seems to be driven by capital expenditure on compute that is growing even faster.
Physical illness may seem like an overreaction about an AI company’s use of AI, and it may well be. But the worldview that produces such reactions is based on fear of “recursive self-improvement”—when AI gets smart enough to make even better AI, which then can make even better AI, and so on—as the precursor to inevitable human disempowerment. In broad strokes, the people with that worldview have gotten an awful lot about AI right so far—so no one should dismiss their reactions out of hand.
At the same time, I am concerned that the conversation about recursive self-improvement (RSI) has become unfocused, extreme, and dramatically oversimplified. It reminds me of how, during my lifetime, the climate movement morphed from one focused on carbon emissions to a broader anti-growth movement that tended to pathologize energy use in general, despite the fact that energy use per capita is one of the surest correlates of human wealth, wellbeing, and health. In similar ways, and echoing Scott Alexander’s own recent critique of how the AI safety movement is evolving, I think there is a risk that fear of uncontrollable RSI is leading to policies that pathologize the use of AI to improve AI (a longstanding practice in the field, to be clear), ban AI progress in crucial domains like biomedicine, and ignore very real tradeoffs America faces with respect to economic growth, geopolitical competition, and national security.
Two Kinds of Recursion
Let’s start by making one thing clear: allowing AI to control the development of its successors with no or little practical human oversight—essentially, handing over the keys of America’s largest AI companies to AI—is a bad idea. No matter how useful AI is for accelerating AI R&D, humans should remain in control, not just in name but in actual fact, over the development of AI. No amount of technological progress is worth the forfeiture of practical human control over AI companies, or, for that matter, of other human institutions. I support the development of laws, regulations, and norms to enforce this worldwide.
But this risk—a relatively narrow one—isn’t what current conversations about RSI focus on in practice. To understand what I mean, it’s helpful to think of two distinct ways in which a technology might recursively improve itself. I’ll call these weak recursion and strong recursion.
Weak recursion is when a technology is a key input into the production of more and better versions of that same technology. It is an extremely common pattern in technology development—really it is the historical norm rather than an aberration. Iron Age metalsmiths used iron as one of the key inputs to make more iron, and stronger alloys of it too. Railroads were used to transport the equipment and materials needed to lay down yet more railroads. Software engineers use software to make new, improved, and more secure software. And in AI, first as an academic field and then as a large industrial enterprise, AI has been a major input to the development of new and better AI for years now. It is easier to think of technologies that are accelerated by weak recursion than it is to think of technologies that are not.
Weak recursion has been a crucial driver of human technological progress over centuries and millennia, and in this sense there is nothing especially weak about it. In absolute terms, it is one of the strongest dynamics in technological history. Weak recursion is only weak relative to strong recursion.
Strong recursion is when a technology can autonomously plan and execute the development of its successor. As far as we know, strong recursion has never happened at any point in history. Even to the extent one views the human brain as a technology (which even I, as a person with an expansive conception of technology, do not), it is not the case that humans are capable of strong recursion. Each human generation adds to the foundation of our civilization’s knowledge and capabilities, which gives the next generation a stronger base from which to build—but we do not design the brains of our children to be superior to our own.
Strong recursion would be a genuinely new thing under the sun. We should take this possibility seriously, but one part of taking a thing seriously is to adhere to rigorous epistemic discipline in analysis. Thus, one should not jump to the conclusion that “recursive self-improvement is happening” if that recursion is of the weak variety instead of the strong.
Who is Doing the Specifying?
The METR study of Anthropic’s internal R&D acceleration is undeniably good evidence that weak recursion is taking place. But is it evidence that strong recursion is underway? It depends how you look at it. METR’s finding is specifically that Anthropic has doubled the rate at which its models’ capabilities improve on well-specified tasks, which makes one wonder who, exactly, is doing the specifying. Very often, I suspect, it is tasks specified by human researchers, based on high-level, firm-wide objectives decided by human managers and executives. If so, that looks more like weak recursion than it does like strong recursion. I see almost no evidence, either from the METR study or from public disclosures of research-acceleration metrics by OpenAI or Anthropic, that strong recursion is taking place. I see ample evidence from those sources that weak recursion is taking place at a large and quickly growing scale.
METR says it is confident that the acceleration they have measured is not due to scaling training compute, but training compute is not the only compute that matters. Anthropic’s software engineering, experiment design, synthetic data generation, and analysis run on inference compute, which every frontier lab has scaled enormously since the report’s baseline period of March 2024 to October 2025. These tasks constitute much of what we refer to as “AI R&D.” This is precisely what weak recursion looks like: a general-purpose technology serving as an input to its own improvement, bounded by the capital its builders can marshal.
Now, as I said, weak recursion is not really weak: it is one of the most powerful dynamics in human history, rhyming if not synonymous with the concept of compounding growth. In the context of AI, weak recursion could result in a rapid increase in frontier AI capabilities—more rapid than anything we have seen so far—over the coming year or so. It could significantly raise the risk profile of the top AI systems, including in domains with the potential to do large-scale damage such as cryptography, biomedicine, and chemicals design.
All this is good reason to require the largest AI companies to embed third-party evaluators inside their development teams, publish disclosures of concerning incidents of misalignment or loss of control (even if those incidents were “warning shots” that did not cause material harm), maintain rigorous cybersecurity, be transparent about their safety practices, and undergo audits for compliance with those safety practices—each of which is a policy I’ve supported.
But do the risks of weak recursion justify banning the practice of using AI as an input to make more and better AI? In my view, they do not. Each of the risks I have highlighted is just one side of the coin: better chemicals design might yield new materials or better industrial processes for producing existing chemicals. Superior capabilities in synthetic biology are essential to ushering in the revolution in biomedicine that I and other AI observers believe is eminently achievable in the near term. AI systems that can find vulnerabilities in software, or weaknesses in cryptography, are simply discovering weaknesses that already existed, ready to exploit at any time (and, plausibly, were already known to large nation-states, including those adversarial to the U.S. and its allies).
What’s more, weak recursion with AI can be used to make AI systems safer. Every frontier AI company’s AI safety strategy relies upon the use of increasingly capable AI systems for a panoply of tasks, including monitoring agents to assess malfeasance or loss of control, using AI to improve internal cybersecurity, and using AI for research in fields like interpretability and alignment (which, in addition to being areas that can improve safety, can also improve capabilities). The path forward for AI safety would become far rockier, perhaps impossible, without AI as a tool to massively augment human effort. For these reasons, a ban on “RSI” that includes weak recursion could easily have the effect of banning most of frontier AI development in practice.
The Problem of Taste
What about strong recursion, however? Is it a real risk, and if we could somehow isolate strong recursion as a legal matter, should we ban that?
My view is that strong recursion is a genuine risk, possibly in the near term, though the evidence for it is thinner than is sometimes supposed. Strong recursion requires AI to take on AI research and development end-to-end—noticing problems, navigating the fraught path between nebulous problems and sharpened questions (often the most challenging part of any intellectual pursuit), ideating solutions, designing experiments, executing experiments, and integrating improvements into technical infrastructure that, in the case of frontier AI companies, serves millions or even billions of people per day. And it requires AI that can do all those things not in some abstract space, but in the real world, where information is imperfect and resources are scarce. Complex tradeoffs about how to allocate capital (compute) between numerous different promising research directions, today made at the highest levels of AI companies, would also have to be made wholly autonomously by AIs.
It is true that AI improves rapidly, but much of what I have described above has remained elusive to even the best AI systems. Some of this work requires taste and discernment—incredibly challenging to measure objectively—and the ability to sharpen those skills over years. Many of the most senior scientists in the field of deep learning, which undergirds all of modern AI, got into the field years or decades before it bore fruit. An empirical evaluation of their research taste would have shown these scientists to have quite poor taste, and quite low research productivity—right up until the deep learning revolution began in 2012. Some AI training environments involve models working for days or more today, but years? It is unclear how we would even design evaluations for long-range intellectual taste.
It was far easier to look at the capabilities of 2020-vintage AI systems like GPT-3 and predict that they would soon surpass humans in domains like cybersecurity. Even the rudimentary capabilities of those older systems put the writing on the wall, and by the time of GPT-4—which could reliably write simple software programs—it was clear that models would accelerate to very high levels of coding ability. Models, by that point and at every point thereafter, made reliable progress in coding month after month.
Progress in “taste,” on the other hand, while nonzero, has been small by comparison. I have used the best AI systems I could get my hands on almost literally every day for four years, and while I have been augmented very greatly, and my day-to-day work has been transformed, I still feel that it is very much me shaping the direction of my research. The ideas that AI systems suggest for open-ended research have improved in the past four years, but they still lack either originality, promise, or both.
It is not obvious to me that models becoming yet more superhuman in mathematics, or superhuman at drug design, or at composing Bach-like fugues as well or better than Bach himself, would really change this dynamic on their own. The questions “can the model do this [even halfway specified thing] well?” and “does the model know what to do next in open-ended intellectual exploration?” are distinct from one another, and this distinction should not be dismissed—as it too often is—as “insufficiently AGI-pilled.” The reality is we do not know how much this distinction matters, and do not know how long it will take for models to master “taste” in open-ended intellectual domains.
METR’s report is consistent with this, stating “we understand that AI is not yet able to make highly significant and fully autonomous contributions to the most important parts of the model training process,” and that internal R&D has been accelerated from AI assisting human researchers. They also indicate that Anthropic employees report “signs of life” on making “highly significant and fully autonomous contributions” to AI R&D, but the details are again redacted. This could be a signal for much better taste to come soon, but it is hard to know what that means—and it is worth noting that every frontier model I’ve tried has had “signs of life” on intellectual taste with respect to its predecessor models.
The evidence we have seen for weak recursion, however compelling it may be, is not by itself especially strong evidence that the leaps needed for strong recursion are right around the corner. But as I have said, strong recursion is both unprecedented and a genuine risk. AI capabilities advance quickly enough, and there is sufficient uncertainty about what further improvement might mean, that it is worth taking the risk of wholly autonomous AI development seriously, including in the near term.
For a while, people speculated that the relative weakness of models in computer use (the ability to manipulate graphical user interfaces) was a deep limitation; it turned out to have been largely solved this year. The same could be true of “taste”; maybe this characteristic we find so ineffable and mysterious will simply fall out of another round of scaling. It is certainly possible, and no one should take anything I’ve written here as an argument that models will never, or will not soon, develop “taste.” I am instead arguing that the case for how we get from “low taste” to “high taste” is not as straightforward as it has been for many of the other domains of AI capability improvement (e.g. cyber). This is why organizations like METR, which try to measure AI research acceleration as carefully as they practically can, are valuable, and why I support laws to bolster the role of such organizations in frontier AI governance.
Governing Strong Recursion
Some would argue that what I’ve called weak recursion, if practiced for long enough, eventually leads to strong recursion, because the capabilities acceleration that comes from weak recursion will yield models capable of strong recursion. Others might say that weak recursion, practiced at sufficient scale, would blur the boundaries with strong recursion in practice. This is especially true in machine learning research and development, since it is today an empirical and search-driven field, meaning that experimentation done at massive scale may be able to serve as a stand-in for taste. Even then, compute is scarce, and deciding which of countless possible experiments deserves it is itself a matter of taste. As the scale of weak recursion within frontier AI companies continues to grow, more transparency about how compute allocation decisions are made, the share of merged code changes with no substantive human review, and similar metrics would be useful. These metrics might be doubly useful in that they could form the basis for a statutory definition of strong recursion.
More broadly, delineating between weak and strong recursion is where it becomes clear that frontier AI governance should be about regulating companies rather than regulating models. Weak recursion can be measured at the level of a model, but strong recursion definitionally concerns issues related to the management of the firm, the capital-allocation decisions made within the firm, and related topics. Strong recursion can therefore only be assessed at the entity level, and this is one of many reasons I believe that independent technical assessment bodies like METR should consider themselves analysts of the frontier AI companies as entities just as much as they consider themselves to be analysts of their models.
It is also worth noting that existing law presents a number of incentives against strong recursion. Corporate law creates duties upon the management of a firm to shareholders—duties which likely would be tested, to say the least, by management choosing to hand over control of the company’s capital equipment to AI with substantial uncertainty and little practical say over what the AI will do with it. Common-law liability creates a default duty of reasonable care upon companies to the rest of society to avoid harms. Consumer-protection law, too, could have a role to play.
All these existing bodies of law could be brought to bear by creative lawyers, and the mere existence of a major lawsuit can be just as much of an impediment for a company as a negative outcome in a lawsuit. Meanwhile, the federal executive has extensive powers at its disposal to hinder frontier AI companies unilaterally (for example, the export controls placed by the Trump Administration on Anthropic’s Fable and Mythos models this past June). No firm contemplating a handover of practical control from humans to machines can ignore this kind of legal and regulatory risk.
While these are legitimate impediments to strong recursion, they are also improvised, unpredictable, and just as easily turned against ordinary and beneficial use of AI. This is why clear, stable, and tailored rules are preferable in this instance to the application of existing law, willy-nilly, to the governance of recursive self-improvement.
Calibration
AI policy would benefit from drawing a sharper distinction between what I’ve called weak and strong recursion. Right now, evidence of weak recursion—which, again, is the norm in the development of technology, especially general-purpose technology—is being used as evidence of strong recursion, while in reality evidence for the latter remains questionable.
Given the increased salience of AI to the public, people who have been influential in the narrow world of AI policy and safety will start to become influential outside of their bubbles. As this occurs, their words will carry more weight. What used to be casual posting on X becomes messages that could impel serious and near-term policy action, or foment panic in the broader public. It is good that AI is becoming more salient, and good that, at least to some extent, policymakers and the public are turning to people with expertise for advice. But as the gravity of that advice evolves, so too must its tenor. This is a transition I, myself, have had to go through with my public communication, and one which I still navigate—unevenly—today. This does not mean one should soften one’s words about the seriousness of AI risk, but it does demand excellent calibration, and at this moment I feel the AI safety and policy disciplines risk becoming as poorly calibrated as I have ever seen them.
The kind of panic that today’s rhetoric could provoke could easily result in laws that delay legitimate and desirable benefits of AI, put the U.S. at a strategic disadvantage on the global stage, and centralize power over AI in the hands of the state. The latter outcome is close to what happened with nuclear technology, where the centralization of power and proliferation of rules impeded the benefits of nuclear energy to consumers, all while states amassed and maintained stockpiles of nuclear weapons strong enough to extinguish human civilization many times over. I continue to believe that risks of this sort are among the likeliest bad AI futures, and worry that this risk is drastically underrated relative to the more traditional projections made by AI safety advocates. We ignore them at our peril, for one thing we can all agree on is that the future, whatever it may bring, is coming on faster than any of us thought.
Thank you for saying this. While it’s true that RSI could impose serious harms like loss of control or extinction, those pale in comparison to the horrors of OpenAI being regulated without the long and thoughtful input of Dean and his wise friends (who all work at OpenAI).